diff --git a/docs/recruitment-information-collection-import-plan.md b/docs/recruitment-information-collection-import-plan.md new file mode 100644 index 0000000..ce1e823 --- /dev/null +++ b/docs/recruitment-information-collection-import-plan.md @@ -0,0 +1,636 @@ +# 招聘信息采集监测分析:外部岗位导入与指纹同步实施计划 + +## 1. 文档目的与范围 + +本文定义以下两部分的完整实施方案: + +1. 在 `/Users/lapuda/code/excel_upload` 建设供外部人员上传岗位 Excel 的独立系统; +2. 在 `shz-backend` 中建设岗位文件的定时拉取、批次导入、岗位指纹比对、增量新增、缺失逻辑删除,以及“招聘信息采集监测分析”菜单能力。 + +本方案以外部 Excel 的**全量快照**为前提。每个成功上传并通过校验的文件代表其声明来源范围内的当前全部有效岗位。 + +### 1.1 本期明确的同步规则 + +外部导入岗位必须带有独立的外部标记,且只能在外部岗位集合内部进行指纹比对: + +```text +上一成功批次的外部岗位指纹集合 = previous +本次已校验、已去重的外部岗位指纹集合 = current + +current ∩ previous:保留原岗位 ID;字段有变化时原地更新 +current - previous:新增外部岗位 +previous - current:逻辑删除外部岗位 +``` + +“删除”采用现有系统惯例的**逻辑删除**,即设置 `del_flag = '2'`,并同步下架岗位;不物理删除记录,不影响内部手工岗位、招聘会岗位或未标记为外部导入的历史岗位。 + +若一个已逻辑删除的外部岗位在后续文件中再次出现,则恢复原岗位记录和原岗位 ID,而不是再创建一条新岗位。 + +### 1.2 不在本期自动执行的事项 + +- 不自动删除或合并既有历史 `job` 数据; +- 不自动删除企业数据;岗位快照缺失只影响外部岗位,不影响 `company`; +- 不将企业名称相近的记录自动视为同一企业; +- 不部署、不重启服务、不执行生产数据库写操作。 + +## 2. 已审查的数据现状 + +审查基于 2026-07-21 的生产 HighGo 数据库只读快照,以及 Excel 样例 `石河子市数据20260714-0720.xlsx`。 + +### 2.1 数据量与关键风险 + +| 对象 | 数量 | 关键结论 | +|---|---:|---| +| `job` | 1,601 | 1,601 条均无 `job_url`,均无 `row_id` | +| 有效 `job` | 1,582 | 269 条未关联 `company_id` | +| `company` | 1,972 | 企业性质全部为空,存在 130 组同名企业 | +| `row_work` | 2,230 | `Aae397`、`Collect_time` 两个日期字段均为空 | +| Excel 样例 | 2,067 | 40 列、6 个来源、岗位链接和采集日期均完整 | + +当前 `job` 表没有岗位链接和原始行关联,不能��于��靠判断新 Excel 中的岗位是否已经存在。当前 `company` 表中有 139 条超出首条的同名企业记录,也不能通过“企业名称取第一条”的方式安全关联。 + +### 2.2 当前库质量指标 + +| 表 | 指标 | 数量 | 比例 | +|---|---|---:|---:| +| `job` | 最低薪资为空 | 303 | 18.93% | +| `job` | 最高薪资为空 | 355 | 22.17% | +| `job` | 薪资区间非法 | 6 | 0.37% | +| `job` | 经纬度任一为空 | 1,595 | 99.63% | +| `job` | 岗位链接为空 | 1,601 | 100% | +| `job` | 原始行 ID 为空 | 1,601 | 100% | +| `company` | 所在地为空 | 928 | 47.06% | +| `company` | 行业为空 | 1,385 | 70.23% | +| `company` | 规模为空 | 1,062 | 53.85% | +| `company` | 企业性质为空 | 1,972 | 100% | +| `company` | 企业简介为空 | 1,498 | 75.96% | + +现有 `job` 表中有 318 组疑似重复岗位键组,涉及 657 条岗位、339 条超出首条的记录;`company` 表有 130 组同名企业,涉及 269 条企业、139 条超出首条的记录。 + +因此,本期外部导入不应将历史库无来源链接的数据纳入“删除比对”。首批导入只比较带外部导入标记、且已具备指纹的岗位。 + +### 2.3 当前 Excel 样例摘要 + +样例工作表为 `合并数据`,共 2,067 行。来源分布如下: + +| `ORG` 来源 | 行数 | +|---|---:| +| 石河子招聘网 | 1,404 | +| 智联招聘 | 541 | +| 天互招聘网 | 79 | +| 猎聘网 | 17 | +| 乌鲁木齐快聘网 | 15 | +| 前程无忧 | 11 | + +样例中有 1,670 个不同的“来源 + 岗位链接”组合,397 行是批内重复组合。导入前必须先按岗位指纹去重,不能把同一岗位重复写入数据库。 + +样例字段质量: + +| Excel 字段 | 非空率 | 导入策略 | +|---|---:|---| +| `Aca112` 岗位名称 | 100% | 强制字段 | +| `Acb22a` 岗位描述 | 100% | 强制字段 | +| `SalaryLow` / `SalaryHight` | 100% | 强制数值和区间校验 | +| `Aae397` 发布日期 | 100% | 解析并保存 | +| `Collect_time` 采集日期 | 100% | 解析并保存 | +| `AAB004` 企业名称 | 100% | 强制字段、企业匹配键 | +| `ORG` 来源 | 100% | 强制字段、来源映射键 | +| `ACE760` 岗位链接 | 100% | 强制字段、优先岗位指纹键 | +| `AAE006` 岗位地址 | 31.79% | 非强制,按区县/城市回退 | +| `Std_class` 岗位分类 | 94.82% | 保存原始分类并做标准化映射 | +| `AAB092` 企业简介 | 32.08% | 有值时补充企业信息 | +| `IndustryType` | 18.82% | 可选行业补充 | +| `IndustrySub` | 5.37% | 可选二级行业补充 | +| `Num_employers` | 99.27% | 企业规模原始值/回退值 | +| `Num_OK` | 26.17% | 企业规模标准值优先 | +| `AAB019_OK` | 26.17% | 企业性质候选值,须映射 | +| `AAE004` 联系人 | 27.72% | 仅补充企业空联系人 | +| `AAE005` 联系电话 | 0% | 本期不能补充联系电话 | + +## 3. 目标架构 + +```text +外部人员 + │ 登录、上传 Excel、查看上传结果 + ▼ +excel_upload 外部上传系统 + │ 文件元数据、SHA-256、校验状态、内部下载接口 + ▼ +shz-backend Quartz 定时任务 + │ 获取最新文件清单 → 下载 → 暂存解析 → 校验/去重 + ▼ +导入批次与原始行暂存 + │ 计算 identity fingerprint 和 content fingerprint + ▼ +外部岗位指纹集合比对 + ├─ 指纹已存在:保留岗位 ID,按内容更新 + ├─ 本次新指纹:新增岗位 + └─ 历史指纹未出现:逻辑删除外部岗位 + ▼ +job / company / ES / 监测数据 + ▼ +来源管理、入库监测、采集趋势、统计分析 +``` + +## 4. 外部岗位特殊标记与指纹设计 + +### 4.1 `job` 表新增字段 + +不复用现有含义不明的 `data_source='5'`,增加专用字段。 + +| 字段 | 类型建议 | 说明 | +|---|---|---| +| `external_import_flag` | `CHAR(1)` | `1` 外部文件导入岗位,`0` 非外部岗位;默认 `0` | +| `external_source_code` | `VARCHAR(64)` | 稳定来源编码,如 `zhaopin`、`51job` | +| `external_source_name` | `VARCHAR(200)` | Excel 原始 `ORG` 名称,用于审计展示 | +| `job_fingerprint` | `VARCHAR(64)` | 岗位身份指纹,SHA-256 十六进制值 | +| `job_content_fingerprint` | `VARCHAR(64)` | 岗位内容指纹,用于判断同一岗位是否需要更新 | +| `external_batch_id` | `VARCHAR(64)` | 最近一次成功见到该岗位的导入批次 | +| `external_first_seen_at` | `TIMESTAMP` | 首次外部导入时间 | +| `external_last_seen_at` | `TIMESTAMP` | 最近一次在成功快照中出现的时间 | + +在历史清洗完成后,增加仅针对有效外部岗位的部分唯一索引: + +```sql +CREATE UNIQUE INDEX uk_job_external_active_fingerprint + ON shz.job (job_fingerprint) + WHERE external_import_flag = '1' AND del_flag = '0'; +``` + +指纹中已经包含来源编码,因此不同来源中相同 URL 或同名岗位不会冲突。 + +### 4.2 岗位身份指纹 + +岗位身份指纹用于判断“这个岗位是否还是同一个岗位”,不应包含经常变化的描述、薪资、采集日期等内容。 + +#### 优先规则:来源 + 规范化岗位链接 + +```text +fingerprint_version = FP_V1 +identity_payload = FP_V1 | external_source_code | normalized_job_url +job_fingerprint = SHA-256(identity_payload) +``` + +`normalized_job_url` 的标准化规则: + +- 删除前后空白; +- URL scheme 和 host 转小写; +- 规范化末尾 `/`; +- 去除已确认无业务含义的推广参数,如 `utm_*`; +- 不删除可能影响岗位身份的路径、岗位 ID 或业务查询参数; +- 解析失败时记录行错误,不静默生成错误指纹。 + +样例所有岗位都具备 `ACE760`,因此正常情况下均使用该规则。 + +#### 降级规则:无岗位链接时 + +只有历史文件或异常数据没有岗位链接时,才允许使用低置信度降级指纹: + +```text +FP_V1 | +external_source_code | +normalized_company_name | +normalized_job_title | +normalized_city_or_county | +salary_low | +salary_high +``` + +降级指纹必须在导入结果中标记为 `LOW_CONFIDENCE`。若同一批次出现多个同指纹但内容不同的行,则阻止自动发布或进入人工确认,不允许任意覆盖。 + +### 4.3 岗位内容指纹 + +身份指纹相同但岗位内容变化时,不应删除并重新创建岗位。应保留原 `job_id`,对允许同步的字段执行原地更新。 + +```text +content_payload = + job_title | description | salary_low | salary_high | + education | experience | vacancies | + job_address | area_code | job_category | + company_name | source_job_url | collect_date + +job_content_fingerprint = SHA-256(canonical_content_payload) +``` + +处理规则: + +| 身份指纹 | 内容指纹 | 动作 | +|---|---|---| +| 已存在 | 相同 | 保留,更新 `external_last_seen_at` 和批次 ID | +| 已存在 | 不同 | 保留 `job_id`,更新外部可同步字段,记录“内容更新” | +| 不存在 | 任意 | 新增外部岗位 | +| 历史存在但已逻辑删除 | 任意 | 恢复原岗位,再按内容更新 | + +### 4.4 删除边界 + +删除比对只能作用于以下集合: + +```text +external_import_flag = '1' +AND del_flag = '0' +AND external_source_code IN 本批次完整来源范围 +``` + +不允许比较或删除: + +- `external_import_flag = '0'` 的内部岗位; +- 当前无来源、无链接、无指纹的历史岗位; +- 招聘会、企业自行发布、后台人工录入等其他业务岗位; +- 不属于本批次文件声明来源范围的外部岗位。 + +逻辑删除时统一设置: + +```text +del_flag = '2' +job_status = '1' +is_publish = 0 +update_by = 'external-import' +update_time = 当前时间 +``` + +删除记录必须保留原 `job_id`、原指纹、最后出现批次和删除批次,支持后续恢复和审计。 + +## 5. 导入批次与数据库设计 + +### 5.1 新增表 + +#### `job_import_batch` + +每次文件导入一条批次记录。 + +| 字段 | 说明 | +|---|---| +| `batch_id` | UUID 或雪花 ID | +| `file_id` | 外部上传系统中的文件版本 ID | +| `file_name` | 原始文件名 | +| `file_sha256` | 文件摘要;成功导入后禁止重复处理 | +| `snapshot_scope` | 本批次覆盖的来源编码集合 | +| `fingerprint_version` | 如 `FP_V1` | +| `status` | `DOWNLOADING`、`VALIDATING`、`STAGING`、`RECONCILING`、`SUCCESS`、`FAILED`、`BLOCKED` | +| `total_rows` | Excel 有效行数 | +| `valid_rows` | 可发布行数 | +| `invalid_rows` | 解析/校验失败行数 | +| `duplicate_rows` | 批内重复行数 | +| `insert_count` | 新增岗位数 | +| `update_count` | 同身份岗位内容更新数 | +| `keep_count` | 指纹和内容都未变的保留数 | +| `restore_count` | 已删除后重新出现、被恢复的岗位数 | +| `delete_count` | 本次逻辑删除岗位数 | +| `error_summary` | 批次级异常摘要 | +| `started_at` / `completed_at` | 执行时间 | + +#### `job_import_row` + +保存每一行的处理结果,支持错误下载、重复定位和审计。 + +| 字段 | 说明 | +|---|---| +| `batch_id` | 所属导入批次 | +| `excel_row_number` | Excel 真实行号 | +| `source_name` / `source_code` | 原始来源与标准来源 | +| `job_fingerprint` | 岗位身份指纹 | +| `content_fingerprint` | 岗位内容指纹 | +| `raw_payload` | 原始字段 JSON 或文本快照 | +| `validation_status` | `VALID`、`INVALID`、`DUPLICATE`、`LOW_CONFIDENCE` | +| `result_status` | `INSERTED`、`UPDATED`、`KEPT`、`RESTORED`、`SKIPPED`、`FAILED` | +| `error_code` / `error_message` | 行错误详情 | +| `job_id` | 最终关联岗位 ID | + +#### 来源别名映射 + +建议新增 `job_import_source_alias`,避免把 Excel 的 `ORG` 字符串直接硬编码在 Java 中。 + +| 字段 | 说明 | +|---|---| +| `alias_name` | Excel 原始 `ORG`,如“智联招聘” | +| `source_code` | 稳定来源编码,如 `zhaopin` | +| `website_id` | 关联 `website_management` | +| `is_active` | 是否允许导入 | +| `snapshot_required` | 全量快照中是否必须存在该来源 | +| `remark` | 映射依据与业务备注 | + +### 5.2 数据迁移原则 + +- 所有 SQL 均采用增量 migration;不执行现有 `job_data_tables.sql` 中的 `DROP TABLE`; +- 先建立批次、暂存和字段,再接通调度; +- 历史岗位默认 `external_import_flag = '0'`,不参与首批删除; +- 先在暂存层完成去重和校验,再写入 `job`、`company`; +- 只有整个发布事务成功,批次才能标为 `SUCCESS`; +- 发布失败时保留上一个成功快照和其可见岗位。 + +## 6. Excel 字段映射与质量策略 + +### 6.1 岗位映射 + +| Excel 字段 | 目标字段 | 规则 | +|---|---|---| +| `Aca112` | `job.job_title` | 强制;去首尾空白;空值拒绝该行 | +| `Acb22a` | `job.description` | 强制;保留全文 | +| `SalaryLow` | `job.min_salary` | 强制;转整数;必须大于等于 0 | +| `SalaryHight` | `job.max_salary` | 强制;必须不小于最低薪资 | +| `Salary` | 原始薪资文本 | 保留在原始行,供审计和异常兜底 | +| `Aae397` | `job.posting_date` | 支持 Excel 日期数值与文本日期 | +| `Collect_time` | 批次原始采集日期 | 支持 Excel 日期数值与文本日期 | +| `Acb240` | `job.vacancies` | 优先解析;为空时回退 `Recruit_Num` | +| `Recruit_Num` | `job.vacancies` | 兜底字段,不覆盖已解析的 `Acb240` | +| `Aac011` | `job.education` | 学历别名标准化后写入字典编码 | +| `Experience` | `job.experience` | 经验别名标准化后写入字典编码 | +| `Std_class` | `job.job_category` | 记录原始分类,并按映射表生成标准分类 | +| `AAE006` | `job.job_address` | 保存详细地址;可为空 | +| `AAE006` / `County` / `City` | `job.job_location` | 按详细地址、区县、城市顺序回退 | +| `County` | `job.job_location_area_code` | 匹配区域字典;不匹配时记录告警 | +| `ORG` | 来源相关字段 | 必须通过来源别名表解析;未知来源阻止发布 | +| `ACE760` | `job.job_url` | 强制;规范化后生成身份指纹 | + +### 6.2 企业映射 + +| Excel 字段 | 目标字段 | 规则 | +|---|---|---| +| `AAB004` | `company.name`、`job.company_name` | 强制;参与企业标准化匹配 | +| `AAE006` / `County` / `City` | `company.location` | 仅补充缺失字段,不覆盖人工维护数据 | +| `AAB092` | `company.description` | 有值时补充;空值不得覆盖已有简介 | +| `IndustrySub` / `IndustryType` | `company.industry` | 二级行业优先;必须通过行业映射或保留原值 | +| `Num_OK` / `Num_employers` | `company.scale` | 标准规模优先;原始人数区间作为回退 | +| `AAB019_OK` / `AAB019` | `company.company_nature` | 通过企业性质别名表映射;未知值待确认 | +| `AAE004` | `company.contact_person` | 仅补充已有空值 | +| `AAE005` | `company.contact_person_phone` | 本样例全空,不更新现有电话 | + +### 6.3 必填、告警和拒绝规则 + +#### 行级拒绝 + +- 岗位名称为空; +- 企业名称为空; +- 来源为空或不在来源映射表; +- 岗位链接为空且无法生成唯一的低置信度降级指纹; +- 最低/最高薪资为空、非数值或最低薪资大于最高薪资; +- 发布日期、采集日期无法解析; +- 同一批次同指纹多条记录且无法判定最新/最完整行。 + +#### 允许导入但记录质量告警 + +- `AAE006` 为空,使用区县或城市回退; +- 区县无法映射到区域字典; +- 学历、经验、企业性质、行业无法映射标准字典; +- 招聘人数无法解析; +- 企业匹配到多个同名企业; +- 无岗位链接导致使用低置信度指纹; +- 经纬度为空,等待异步地理编码。 + +## 7. 全量快照导入流程 + +### 7.1 定时任务入口 + +沿用 RuoYi Quartz 的任务管理机制,新增可由 `sys_job` 调用的入口,例如: + +```text +jobImportCron.importLatestExternalSnapshot() +``` + +默认延续旧逻辑的周三、周六凌晨 03:00 调度;实际 cron 表达式由 `sys_job` 初始化数据配置,并允许后台修改。迁移脚本会先以“暂停”状态创建任务,只有完成外部服务 URL、内部令牌和来源映射配置后才由管理员启用。任务必须设置为不允许并发执行。 + +### 7.2 流程步骤 + +1. 获取外部系统的最新文件元数据:文件 ID、上传时间、SHA-256、来源范围、下载地址; +2. 若同一 SHA-256 已有成功批次,直接跳过,记录“重复文件”; +3. 获取分布式锁,防止多节点和人工触发并发导入; +4. 下载至临时目录,校验扩展名、真实文件类型、文件大小、工作表和表头; +5. 用流式解析器读取 Excel,逐行保存到暂存记录; +6. 解析来源别名、日期、薪资、学历、经验、区域、企业字段; +7. 计算岗位身份指纹和内容指纹; +8. 先做批内指纹去重:同一指纹保留采集时间更新且字段最完整的记录; +9. 校验批次覆盖范围。若要求全量文件包含的来源缺失、有效行数为 0、删除比例异常高,则将批次标记为 `BLOCKED`,不发布; +10. 在同一个发布事务中执行岗位比对、新增、更新、恢复和逻辑删除; +11. 更新企业的可补充字段,但不删除企业、不盲目覆盖人工维护字段; +12. 写入 `job_import_batch`、`job_import_row`、入库监测明细和趋势统计; +13. 提交事务后,异步刷新 ES、岗位分类缓存、岗位名称联想缓存; +14. 任务日志记录批次 ID、文件摘要、各类统计和异常摘要。 + +### 7.3 集合比对伪代码 + +```text +current = staged_valid_rows + .deduplicate_by(job_fingerprint) + .where(source_code in snapshot_scope) + +previous = active jobs + .where(external_import_flag = '1') + .where(external_source_code in snapshot_scope) + .index_by(job_fingerprint) + +for row in current: + if previous contains row.job_fingerprint: + keep existing job_id + if content fingerprint changed: + update external-synchronized fields + set last_seen_batch_id = current batch + remove fingerprint from previous + else if logically-deleted external job exists for fingerprint: + restore original job_id and update fields + else: + insert a new external job + +for each remaining job in previous: + logically delete job +``` + +### 7.4 删除保护阈值 + +为防止来源异常、错误文件或空文件导致批量下线,发布前必须执行保护规则: + +- 当前有效行数为 0:禁止发布; +- 必填来源范围不完整:禁止发布; +- 批内错误率超过可配置阈值:禁止发布; +- 本次拟删除外部岗位比例超过可配置阈值(建议默认 30%):标记 `BLOCKED`,要求管理员确认; +- 同一来源数据量相较上一成功批次异常下降时:生成预警并可阻止发布; +- 未成功完成批次时,绝不执行删除阶段。 + +## 8. 外部 Excel 上传系统 + +### 8.1 部署目录与技术边界 + +在 `/Users/lapuda/code/excel_upload` 建立独立服务。推荐使用 FastAPI、持久化文件元数据和本地/对象存储上传目录;其职责仅限于文件接收、校验、保管和安全提供下载,不直接访问业务数据库。 + +### 8.2 页面功能 + +- 登录; +- 上传 Excel; +- 显示模板要求、校验结果、来源范围和预计有效行数; +- 查看历史上传记录; +- 查看文件是否已被后端导入、导入批次状态和错误摘要; +- 下载原始文件或错误报告; +- 管理员可作废错误上传文件,但不可修改已成功导入文件内容。 + +### 8.3 外部系统接口 + +| 接口 | 访问者 | 说明 | +|---|---|---| +| `POST /api/v1/uploads` | 外部登录用户 | 上传并基础校验 Excel | +| `GET /api/v1/files` | 外部登录用户 | 查询本人/授权范围内历史文件 | +| `GET /api/v1/files/{fileId}` | 外部登录用户 | 查询文件元数据和校验状态 | +| `GET /api/v1/files/{fileId}/download` | 外部登录用户 | 下载自己的原始文件 | +| `GET /api/internal/latest` | `shz-backend` | 返回最新可导入文件的元数据 | +| `GET /api/internal/files/{fileId}/download` | `shz-backend` | 下载指定版本文件流 | +| `GET /health` | 监控 | 健康检查 | + +内部接口使用独立的 Bearer Token 或 HMAC 签名;令牌、登录密码散列、上传大小限制和文件目录全部通过环境变量或安全配置管理,不写死在代码、Dockerfile 或前端页面。 + +### 8.4 文件安全要求 + +- 仅接收 `.xlsx`; +- 校验文件签名和 ZIP 结构,不只依赖扩展名; +- 设置上传大小、解压后大小和行数上限; +- 拒绝加密文件、宏文件和异常压缩比文件; +- 文件以服务端 UUID 命名,原始文件名仅作显示; +- 保存 SHA-256;相同摘要文件不重复产生待导入版本; +- 保存上传人、上传时间、来源范围和校验结果; +- 上传文件目录使用独立 volume,不能暴露为静态目录浏览。 + +## 9. 招聘信息采集监测分析菜单 + +父菜单:`招聘信息采集监测分析` + +子菜单使用四个汉字名称: + +```text +招聘信息采集监测分析 +├─ 来源管理 +├─ 入库监测 +├─ 采集趋势 +└─ 统计分析 +``` + +### 9.1 来源管理 + +复用并扩展现有 `website_management`: + +- 来源名称、来源编码、官网地址、所属单位、启停状态; +- Excel `ORG` 别名映射; +- 是否必须出现在全量快照中; +- 最近上传时间、最近成功导入时间、最近数据量; +- 禁止匿名访问,按后台权限管理。 + +### 9.2 入库监测 + +以 `job_import_batch` 和 `job_import_row` 为准展示: + +- 文件名、摘要、来源范围、任务状态; +- 总行数、有效行、错误行、批内重复行; +- 新增、更新、保留、恢复、逻辑删除数量; +- 各来源成功/失败/重复明细; +- 错误行下载; +- 管理员重试、作废、确认高删除比例批次; +- 对接已有 `JobDataStorageDetection`、`JobDataStorageDetail` 数据结构,自动生成而非手工录入。 + +### 9.3 采集趋势 + +展示 7/30/90 天及自定义时间范围内: + +- 文件上传数量; +- 有效原始行数量; +- 新增、更新、保留、恢复、删除、失败、重复数; +- 全部来源总量和单来源趋势; +- 来源数据量环比、异常下降预警; +- 扩展当前仅 `total` 的趋势查询,支持 `source_code` 维度。 + +### 9.4 统计分析 + +基于成功发布的外部岗位和现有岗位统计: + +- 行业、薪资、区域、学历、经验、岗位分类分布; +- 外部来源筛选与时间范围筛选; +- 地址缺失率、行业未映射率、学历/经验未映射率; +- 来源质量对比; +- 与现有 `StaticsController` 接口衔接,但生成统计数据的操作只能由任务或管理员触发。 + +## 10. 权限与安全 + +建议新增权限: + +| 权限 | 说明 | +|---|---| +| `cms:recruit:source:list` | 查询来源 | +| `cms:recruit:source:edit` | 维护来源和别名 | +| `cms:recruit:monitor:list` | 查询导入批次和明细 | +| `cms:recruit:monitor:retry` | 重试失败批次 | +| `cms:recruit:monitor:confirm` | 确认高风险删除批次 | +| `cms:recruit:trend:list` | 查询趋势 | +| `cms:recruit:analysis:list` | 查询统计 | +| `cms:recruit:import:manual` | 手动触发一次外部拉取 | + +需要收口或替换当前的风险入口: + +- 不再向外部暴露按服务器路径导入文件的接口; +- 现有导入接口不能在上传成功后直接清空 `row_work`; +- 来源管理和入库监测接口不得保持 `@Anonymous`; +- 文件下载、手动触发、重试、批次确认均必须留操作日志。 + +## 11. 实施阶段 + +### 阶段一:规则与数据准备 + +1. 确认 6 个 Excel 来源与来源管理表的映射; +2. 确认全量快照的来源范围和缺失来源处理规则; +3. 确认学历、经验、企业性质、规模、行业别名表; +4. 确认企业同名歧义的人工处理策略; +5. 以样例文件形成正式模板和表头版本规则。 + +### 阶段二:外部上传系统 + +1. 创建 FastAPI 项目、登录、上传、文件元数据和文件存储; +2. 实现 Excel 基础校验、SHA-256 和文件历史; +3. 实现后端内部元数据与下载接口; +4. 编写上传、权限、重复文件和非法文件测试。 + +### 阶段三:后端批次导入 + +1. 新增数据库 migration、批次表、原始行表和来源别名表; +2. 扩展 `job` 外部标记和指纹字段; +3. 实现流式 Excel 解析、日期转换、字段规范化和行错误记录; +4. 实现企业匹配、岗位身份指纹和内容指纹; +5. 实现新增、更新、保留、恢复、逻辑删除的发布事务; +6. 实现 Quartz 拉取任务、分布式锁和保护阈值; +7. 在成功提交后刷新 ES 和缓存。 + +### 阶段四:监测与菜单 + +1. 自动写入入库监测、来源明细和趋势统计; +2. 扩展四个后台接口与权限 SQL; +3. 对接后台前端四个页面; +4. 增加错误报告下载、手工重试和异常确认操作。 + +### 阶段五:验证与上线准备 + +1. 用样例 2,067 行文件验证解析、去重和发布结果; +2. 验证 397 个批内重复组合不会产生重复岗位; +3. 验证身份相同、内容改变时保留 `job_id` 并更新内容; +4. 验证新指纹新增岗位; +5. 验证旧指纹在当前快照消失时仅逻辑删除外部岗位; +6. 验证历史内部岗位完全不受删除比对影响; +7. 验证异常文件、缺失来源、空文件、高删除比例会阻止发布; +8. 验证重试、断点恢复、重复文件跳过和 ES 刷新。 + +## 12. 验收标准 + +完成后必须满足以下标准: + +1. 外部人员可登录上传符合模板的 `.xlsx` 文件,并可查看文件校验状态; +2. 后端能安全获取最新文件元数据和文件流,重复文件不会重复导入; +3. 每个成功导入文件都有唯一批次记录、SHA-256、来源范围和统计结果; +4. 每条外部岗位都有 `external_import_flag = '1'`、来源编码、岗位链接、身份指纹和最近批次; +5. 本次与上次指纹相同的岗位保留同一个 `job_id`; +6. 本次新增指纹新增岗位; +7. 上一快照存在、本次快照不存在的外部岗位被逻辑删除; +8. 已删除外部岗位重新出现时恢复原 `job_id`; +9. 内部岗位、手工岗位、招聘会岗位和历史无标记岗位不会被外部快照删除; +10. 监测、趋势、统计四个菜单可查询与导入批次一致的数据; +11. 批内重复、字段错误、来源缺失和高比例删除均可追踪、可下载、可阻止发布; +12. 不执行物理删除,不执行未授权的部署或生产服务重启。 + +## 13. 当前需要业务确认的事项 + +1. “石河子招聘网”是否可视为现有“石河子人才网”的别名,还是必须新建独立来源; +2. 六个样例来源是否为每个全量文件都必须出现的固定来源范围; +3. 对高删除比例批次,默认是完全阻止发布,还是允许管理员在后台确认后继续; +4. 企业同名且地区也无法区分时,是创建待确认企业、沿用既有企业,还是禁止该行岗位发布; +5. 当前历史 `job.data_source = '5'` 的真实来源含义,是否存在旧系统文档可供补充。 diff --git a/ruoyi-admin/src/main/resources/application.yml b/ruoyi-admin/src/main/resources/application.yml index 9d56761..3010c5e 100644 --- a/ruoyi-admin/src/main/resources/application.yml +++ b/ruoyi-admin/src/main/resources/application.yml @@ -123,6 +123,23 @@ mybatis-plus: file: upload-dir: /opt/service/file/data/file +# 外部岗位 Excel 全量快照同步。真实 token 只允许在环境专用配置或密钥管理系统中设置。 +external-job-import: + enabled: false + latest-metadata-url: + internal-token: 8124757d8c37881b4b3d86c60332ca97aa97e6fd6a68db595bb38a1c72bc19dc + connect-timeout-millis: 10000 + read-timeout-millis: 120000 + max-invalid-row-ratio: 0.05 + max-delete-ratio: 0.30 + required-sources: + - shz_job_net + - zhaopin + - tianhu_job + - liepin + - wlmq_fast_job + - 51job + # 政策文件上传配置 policy: upload: diff --git a/ruoyi-bussiness/src/main/java/com/ruoyi/cms/controller/cms/ExternalJobImportController.java b/ruoyi-bussiness/src/main/java/com/ruoyi/cms/controller/cms/ExternalJobImportController.java new file mode 100644 index 0000000..33aa820 --- /dev/null +++ b/ruoyi-bussiness/src/main/java/com/ruoyi/cms/controller/cms/ExternalJobImportController.java @@ -0,0 +1,67 @@ +package com.ruoyi.cms.controller.cms; + +import com.ruoyi.cms.externalimport.ExternalJobImportResult; +import com.ruoyi.cms.externalimport.ExternalJobSourceAliasRequest; +import com.ruoyi.cms.externalimport.IExternalJobImportService; +import com.ruoyi.common.annotation.Log; +import com.ruoyi.common.core.controller.BaseController; +import com.ruoyi.common.core.domain.AjaxResult; +import com.ruoyi.common.enums.BusinessType; +import io.swagger.annotations.Api; +import io.swagger.annotations.ApiOperation; +import org.springframework.security.access.prepost.PreAuthorize; +import org.springframework.web.bind.annotation.GetMapping; +import org.springframework.web.bind.annotation.PostMapping; +import org.springframework.web.bind.annotation.RequestMapping; +import org.springframework.web.bind.annotation.RequestBody; +import org.springframework.web.bind.annotation.RequestParam; +import org.springframework.web.bind.annotation.RestController; + +import java.util.List; +import java.util.Map; + +/** 外部岗位导入批次监控和管理员手动触发接口。 */ +@RestController +@RequestMapping("/cms/externalJobImport") +@Api(tags = "后台:外部岗位导入") +public class ExternalJobImportController extends BaseController { + private final IExternalJobImportService externalJobImportService; + + public ExternalJobImportController(IExternalJobImportService externalJobImportService) { + this.externalJobImportService = externalJobImportService; + } + + @GetMapping("/batches") + @ApiOperation("查询外部岗位导入批次") + @PreAuthorize("@ss.hasPermi('cms:recruit:monitor:list')") + public AjaxResult batches(@RequestParam(required = false) String status, + @RequestParam(required = false) Integer limit) { + List> batches = externalJobImportService.listBatches(status, limit); + return success(batches); + } + + @PostMapping("/run") + @ApiOperation("手动拉取并导入最新外部岗位 Excel") + @PreAuthorize("@ss.hasPermi('cms:recruit:import:manual')") + @Log(title = "手动导入外部岗位 Excel", businessType = BusinessType.IMPORT) + public AjaxResult run() { + ExternalJobImportResult result = externalJobImportService.importLatest(); + return success(result); + } + + @GetMapping("/sources") + @ApiOperation("查询外部 Excel 来源别名") + @PreAuthorize("@ss.hasPermi('cms:recruit:source:list')") + public AjaxResult sources() { + return success(externalJobImportService.listSourceAliases()); + } + + @PostMapping("/sources") + @ApiOperation("新增或修改外部 Excel 来源别名") + @PreAuthorize("@ss.hasPermi('cms:recruit:source:edit')") + @Log(title = "维护外部岗位来源别名", businessType = BusinessType.UPDATE) + public AjaxResult saveSource(@RequestBody ExternalJobSourceAliasRequest request) { + externalJobImportService.saveSourceAlias(request); + return success(); + } +} diff --git a/ruoyi-bussiness/src/main/java/com/ruoyi/cms/controller/cms/JobDataStorageDetectionController.java b/ruoyi-bussiness/src/main/java/com/ruoyi/cms/controller/cms/JobDataStorageDetectionController.java index 4cd1c51..91cc21c 100644 --- a/ruoyi-bussiness/src/main/java/com/ruoyi/cms/controller/cms/JobDataStorageDetectionController.java +++ b/ruoyi-bussiness/src/main/java/com/ruoyi/cms/controller/cms/JobDataStorageDetectionController.java @@ -2,7 +2,6 @@ package com.ruoyi.cms.controller.cms; import com.ruoyi.cms.domain.JobDataStorageDetection; import com.ruoyi.cms.service.JobDataStorageDetectionService; -import com.ruoyi.common.annotation.Anonymous; import com.ruoyi.common.annotation.Log; import com.ruoyi.common.core.controller.BaseController; import com.ruoyi.common.core.domain.AjaxResult; @@ -10,6 +9,7 @@ import com.ruoyi.common.core.page.TableDataInfo; import io.swagger.annotations.Api; import io.swagger.annotations.ApiOperation; import org.springframework.beans.factory.annotation.Autowired; +import org.springframework.security.access.prepost.PreAuthorize; import org.springframework.web.bind.annotation.*; import java.util.List; @@ -20,7 +20,6 @@ import static com.ruoyi.common.enums.BusinessType.QUERY; @RestController @RequestMapping(value = "/cms/storageDetection") @Api(tags = "后台:岗位数据入库检测") -@Anonymous public class JobDataStorageDetectionController extends BaseController { @Autowired @@ -29,6 +28,7 @@ public class JobDataStorageDetectionController extends BaseController { @Log(title = "查询岗位数据入库监测列表", businessType = QUERY) @ApiOperation("查询岗位数据入库监测列表") @GetMapping(value = "/getList") + @PreAuthorize("@ss.hasPermi('cms:recruit:monitor:list')") public TableDataInfo getList(JobDataStorageDetection jobDataStorageDetection) { startPage(); List list = jobDataStorageDetectionService.getList(jobDataStorageDetection); @@ -38,6 +38,7 @@ public class JobDataStorageDetectionController extends BaseController { @Log(title = "查询岗位数据入库监测详情", businessType = QUERY) @ApiOperation("查询岗位数据入库监测详情") @GetMapping(value = "/getSingle/{detectionId}") + @PreAuthorize("@ss.hasPermi('cms:recruit:monitor:list')") public AjaxResult getSingle(@PathVariable String detectionId) { return AjaxResult.success(jobDataStorageDetectionService.getSingle(detectionId)); } @@ -45,6 +46,7 @@ public class JobDataStorageDetectionController extends BaseController { @Log(title = "保存岗位数据入库监测记录", businessType = INSERT) @ApiOperation("保存岗位数据入库监测记录") @PostMapping(value = "/save") + @PreAuthorize("@ss.hasPermi('cms:recruit:monitor:edit')") public AjaxResult save(@RequestBody JobDataStorageDetection jobDataStorageDetection) { try { jobDataStorageDetectionService.saveData(jobDataStorageDetection); diff --git a/ruoyi-bussiness/src/main/java/com/ruoyi/cms/controller/cms/JobDataTrendController.java b/ruoyi-bussiness/src/main/java/com/ruoyi/cms/controller/cms/JobDataTrendController.java index 25bf13e..8efcf5e 100644 --- a/ruoyi-bussiness/src/main/java/com/ruoyi/cms/controller/cms/JobDataTrendController.java +++ b/ruoyi-bussiness/src/main/java/com/ruoyi/cms/controller/cms/JobDataTrendController.java @@ -9,6 +9,7 @@ import com.ruoyi.common.enums.BusinessType; import io.swagger.annotations.Api; import io.swagger.annotations.ApiOperation; import org.springframework.beans.factory.annotation.Autowired; +import org.springframework.security.access.prepost.PreAuthorize; import org.springframework.web.bind.annotation.GetMapping; import org.springframework.web.bind.annotation.RequestMapping; import org.springframework.web.bind.annotation.RequestParam; @@ -29,13 +30,14 @@ public class JobDataTrendController extends BaseController { private IJobDataTrendService jobDataTrendService; /** - * 查询全网采集趋势列表 (website_name = "total") + * 查询全网或单来源采集趋势列表。sourceCode 为空时查询全网(website_name = "total")。 */ @GetMapping("/list") @ApiOperation("获取全网采集趋势数据(Total)") @Log(title = "获取全网采集趋势数据(Total)", businessType = BusinessType.QUERY) + @PreAuthorize("@ss.hasPermi('cms:recruit:trend:list')") public AjaxResult list(@RequestParam Map params) { List list = jobDataTrendService.selectTrendTotalList(params); return success(list); } -} \ No newline at end of file +} diff --git a/ruoyi-bussiness/src/main/java/com/ruoyi/cms/controller/cms/WebsiteManagementController.java b/ruoyi-bussiness/src/main/java/com/ruoyi/cms/controller/cms/WebsiteManagementController.java index dba953f..433ba13 100644 --- a/ruoyi-bussiness/src/main/java/com/ruoyi/cms/controller/cms/WebsiteManagementController.java +++ b/ruoyi-bussiness/src/main/java/com/ruoyi/cms/controller/cms/WebsiteManagementController.java @@ -2,7 +2,6 @@ package com.ruoyi.cms.controller.cms; import com.ruoyi.cms.domain.WebsiteManagement; import com.ruoyi.cms.service.WebsiteManagementService; -import com.ruoyi.common.annotation.Anonymous; import com.ruoyi.common.annotation.Log; import com.ruoyi.common.core.controller.BaseController; import com.ruoyi.common.core.domain.AjaxResult; @@ -10,6 +9,7 @@ import com.ruoyi.common.core.page.TableDataInfo; import io.swagger.annotations.Api; import io.swagger.annotations.ApiOperation; import org.springframework.beans.factory.annotation.Autowired; +import org.springframework.security.access.prepost.PreAuthorize; import org.springframework.web.bind.annotation.*; import org.springframework.web.multipart.MultipartFile; @@ -20,7 +20,6 @@ import static com.ruoyi.common.enums.BusinessType.*; @RestController @RequestMapping("/cms/website") @Api(tags = "后台:岗位信息来源管理") -@Anonymous public class WebsiteManagementController extends BaseController { @Autowired @@ -29,6 +28,7 @@ public class WebsiteManagementController extends BaseController { @Log(title = "查询岗位信息来源网站列表", businessType = QUERY) @ApiOperation("查询岗位信息来源网站列表") @GetMapping(value = "/getList") + @PreAuthorize("@ss.hasPermi('cms:recruit:source:list')") public TableDataInfo getList(WebsiteManagement websiteManagement) { startPage(); List list = websiteManagementService.getList(websiteManagement); @@ -38,6 +38,7 @@ public class WebsiteManagementController extends BaseController { @Log(title = "查询岗位信息来源网站详情", businessType = QUERY) @ApiOperation("查询岗位信息来源网站详情") @GetMapping(value = "/getSingle/{websiteId}") + @PreAuthorize("@ss.hasPermi('cms:recruit:source:list')") public AjaxResult getSingle(@PathVariable Long websiteId) { return AjaxResult.success(websiteManagementService.getById(websiteId)); } @@ -45,6 +46,7 @@ public class WebsiteManagementController extends BaseController { @Log(title = "岗位信息来源网站下拉选", businessType = QUERY) @ApiOperation("岗位信息来源网站下拉选") @GetMapping(value = "/getSelectOptions") + @PreAuthorize("@ss.hasPermi('cms:recruit:source:list')") public AjaxResult getSelectOptions() { WebsiteManagement websiteManagement = new WebsiteManagement(); websiteManagement.setDelFlag("0"); @@ -54,6 +56,7 @@ public class WebsiteManagementController extends BaseController { @Log(title = "保存岗位信息来源网站信息", businessType = INSERT) @ApiOperation("保存岗位信息来源网站信息") @PostMapping(value = "/save") + @PreAuthorize("@ss.hasPermi('cms:recruit:source:edit')") public AjaxResult save(@RequestBody WebsiteManagement websiteManagement) { return toAjax(websiteManagementService.save(websiteManagement)); } @@ -61,6 +64,7 @@ public class WebsiteManagementController extends BaseController { @Log(title = "修改岗位信息来源网站信息", businessType = UPDATE) @ApiOperation("修改岗位信息来源网站信息") @PostMapping(value = "/update") + @PreAuthorize("@ss.hasPermi('cms:recruit:source:edit')") public AjaxResult update(@RequestBody WebsiteManagement websiteManagement) { return toAjax(websiteManagementService.updateById(websiteManagement)); } @@ -68,6 +72,7 @@ public class WebsiteManagementController extends BaseController { @Log(title = "删除岗位信息来源网站信息", businessType = DELETE) @ApiOperation("删除岗位信息来源网站信息") @PostMapping(value = "/delete") + @PreAuthorize("@ss.hasPermi('cms:recruit:source:edit')") public AjaxResult delete(@RequestBody WebsiteManagement websiteManagement) { websiteManagement.setDelFlag("2"); return toAjax(websiteManagementService.removeById(websiteManagement)); @@ -76,6 +81,7 @@ public class WebsiteManagementController extends BaseController { @Log(title = "导入岗位信息来源网站信息", businessType = IMPORT) @ApiOperation("导入岗位信息来源网站信息") @PostMapping(value = "/import") + @PreAuthorize("@ss.hasPermi('cms:recruit:source:edit')") public AjaxResult importData(@RequestParam(value = "file") MultipartFile file,@RequestParam(value = "dataType")String dataType) throws Exception { return websiteManagementService.importData(file,dataType); } diff --git a/ruoyi-bussiness/src/main/java/com/ruoyi/cms/cron/JobCron.java b/ruoyi-bussiness/src/main/java/com/ruoyi/cms/cron/JobCron.java index 40f3e2a..811ff46 100644 --- a/ruoyi-bussiness/src/main/java/com/ruoyi/cms/cron/JobCron.java +++ b/ruoyi-bussiness/src/main/java/com/ruoyi/cms/cron/JobCron.java @@ -6,6 +6,7 @@ import com.ruoyi.cms.service.ICompanyService; import com.ruoyi.cms.service.IESJobSearchService; import com.ruoyi.cms.service.IJobService; import com.ruoyi.cms.service.IJobTitleSuggestService; +import com.ruoyi.cms.externalimport.IExternalJobImportService; import com.ruoyi.common.utils.spring.SpringUtils; import org.springframework.beans.factory.annotation.Autowired; import org.springframework.stereotype.Component; @@ -35,4 +36,12 @@ public class JobCron { public void rebuildJobTitleSuggest(){ SpringUtils.getBean(IJobTitleSuggestService.class).rebuildCache(); } + + /** + * 外部 Excel 全量快照同步。 + * Quartz 调用目标:jobCron.importLatestExternalSnapshot() + */ + public void importLatestExternalSnapshot(){ + SpringUtils.getBean(IExternalJobImportService.class).importLatest(); + } } diff --git a/ruoyi-bussiness/src/main/java/com/ruoyi/cms/domain/Job.java b/ruoyi-bussiness/src/main/java/com/ruoyi/cms/domain/Job.java index 4fbf8a8..3679733 100644 --- a/ruoyi-bussiness/src/main/java/com/ruoyi/cms/domain/Job.java +++ b/ruoyi-bussiness/src/main/java/com/ruoyi/cms/domain/Job.java @@ -17,6 +17,7 @@ import io.swagger.annotations.ApiModelProperty; import lombok.Data; import java.math.BigDecimal; +import java.util.Date; import java.util.List; /** @@ -146,6 +147,32 @@ public class Job extends BaseEntity @ApiModelProperty("岗位链接") private String jobUrl; + @ApiModelProperty("是否外部 Excel 导入岗位(0否,1是)") + private String externalImportFlag; + + @ApiModelProperty("外部来源稳定编码") + private String externalSourceCode; + + @ApiModelProperty("外部 Excel 原始来源名称") + private String externalSourceName; + + @ApiModelProperty("外部岗位身份指纹") + private String jobFingerprint; + + @ApiModelProperty("外部岗位内容指纹") + private String jobContentFingerprint; + + @ApiModelProperty("最近一次外部导入批次") + private String externalBatchId; + + @JsonFormat(pattern = "yyyy-MM-dd HH:mm:ss") + @ApiModelProperty("外部岗位首次出现时间") + private Date externalFirstSeenAt; + + @JsonFormat(pattern = "yyyy-MM-dd HH:mm:ss") + @ApiModelProperty("外部岗位最近出现时间") + private Date externalLastSeenAt; + @ApiModelProperty("jobRow对应id") private Long rowId; diff --git a/ruoyi-bussiness/src/main/java/com/ruoyi/cms/externalimport/ExternalJobCandidate.java b/ruoyi-bussiness/src/main/java/com/ruoyi/cms/externalimport/ExternalJobCandidate.java new file mode 100644 index 0000000..c234183 --- /dev/null +++ b/ruoyi-bussiness/src/main/java/com/ruoyi/cms/externalimport/ExternalJobCandidate.java @@ -0,0 +1,64 @@ +package com.ruoyi.cms.externalimport; + +import lombok.Data; + +import java.util.ArrayList; +import java.util.List; + +/** + * Excel 单行在进入数据库前的规范化模型。 + * 字段名不直接暴露给 API,避免将外部 Excel 表头绑定为内部契约。 + */ +@Data +public class ExternalJobCandidate { + private int excelRowNumber; + private String sourceName; + private String sourceCode; + private String jobTitle; + private String description; + private String educationRaw; + private String educationCode; + private String experienceRaw; + private String experienceCode; + private String companyName; + private String location; + private String jobAddress; + private String areaCode; + private String salaryLow; + private String salaryHigh; + private String salaryText; + private String postingDate; + private String collectDate; + private String vacanciesText; + private String jobUrl; + private String jobCategory; + private String companyDescription; + private String industryType; + private String industrySub; + private String companyScale; + private String companyScaleCode; + private String companyNature; + private String companyNatureCode; + private String contactPerson; + private String jobFingerprint; + private String contentFingerprint; + private List errors = new ArrayList<>(); + private List warnings = new ArrayList<>(); + + public boolean isValid() { + return errors.isEmpty(); + } + + public int completenessScore() { + int score = 0; + String[] values = {description, educationRaw, experienceRaw, location, jobAddress, + vacanciesText, companyDescription, industryType, industrySub, companyScale, + companyNature, contactPerson}; + for (String value : values) { + if (value != null && !value.trim().isEmpty()) { + score++; + } + } + return score; + } +} diff --git a/ruoyi-bussiness/src/main/java/com/ruoyi/cms/externalimport/ExternalJobExcelParser.java b/ruoyi-bussiness/src/main/java/com/ruoyi/cms/externalimport/ExternalJobExcelParser.java new file mode 100644 index 0000000..fc6a8af --- /dev/null +++ b/ruoyi-bussiness/src/main/java/com/ruoyi/cms/externalimport/ExternalJobExcelParser.java @@ -0,0 +1,141 @@ +package com.ruoyi.cms.externalimport; + +import org.apache.poi.ss.usermodel.Cell; +import org.apache.poi.ss.usermodel.DataFormatter; +import org.apache.poi.ss.usermodel.DateUtil; +import org.apache.poi.ss.usermodel.Row; +import org.apache.poi.ss.usermodel.Sheet; +import org.apache.poi.ss.usermodel.Workbook; +import org.apache.poi.ss.usermodel.WorkbookFactory; + +import java.io.IOException; +import java.io.InputStream; +import java.text.SimpleDateFormat; +import java.util.ArrayList; +import java.util.Date; +import java.util.HashMap; +import java.util.List; +import java.util.Map; + +/** + * 解析外部岗位模板。此解析器明确处理 Excel 日期单元格,替代旧导入逻辑中日期全部丢失的行为。 + */ +public class ExternalJobExcelParser { + private static final String[] REQUIRED_HEADERS = { + "Aca112", "Acb22a", "SalaryLow", "SalaryHight", "Aae397", "AAB004", "ORG", "ACE760", "Collect_time" + }; + private final DataFormatter formatter = new DataFormatter(); + private final SimpleDateFormat dateFormat = new SimpleDateFormat("yyyy-MM-dd"); + + public List parse(InputStream inputStream) throws IOException { + try (Workbook workbook = WorkbookFactory.create(inputStream)) { + if (workbook.getNumberOfSheets() == 0) { + throw new IllegalArgumentException("Excel 文件没有工作表"); + } + Sheet sheet = workbook.getSheetAt(0); + Row headerRow = sheet.getRow(0); + if (headerRow == null) { + throw new IllegalArgumentException("Excel 文件缺少表头"); + } + Map headerIndexes = headers(headerRow); + for (String required : REQUIRED_HEADERS) { + if (!headerIndexes.containsKey(required)) { + throw new IllegalArgumentException("Excel 缺少必需表头: " + required); + } + } + List result = new ArrayList<>(); + for (int index = 1; index <= sheet.getLastRowNum(); index++) { + Row row = sheet.getRow(index); + if (row == null || isBlank(row)) { + continue; + } + ExternalJobCandidate candidate = new ExternalJobCandidate(); + candidate.setExcelRowNumber(index + 1); + candidate.setJobCategory(value(row, headerIndexes, "Std_class")); + candidate.setJobTitle(value(row, headerIndexes, "Aca112")); + candidate.setDescription(value(row, headerIndexes, "Acb22a")); + candidate.setEducationRaw(value(row, headerIndexes, "Aac011")); + candidate.setVacanciesText(firstNonBlank(value(row, headerIndexes, "Acb240"), value(row, headerIndexes, "Recruit_Num"))); + candidate.setSalaryText(value(row, headerIndexes, "Salary")); + candidate.setSalaryLow(value(row, headerIndexes, "SalaryLow")); + candidate.setSalaryHigh(value(row, headerIndexes, "SalaryHight")); + candidate.setPostingDate(dateValue(row, headerIndexes, "Aae397")); + candidate.setCompanyName(value(row, headerIndexes, "AAB004")); + candidate.setJobAddress(value(row, headerIndexes, "AAE006")); + candidate.setSourceName(value(row, headerIndexes, "ORG")); + candidate.setJobUrl(value(row, headerIndexes, "ACE760")); + candidate.setContactPerson(value(row, headerIndexes, "AAE004")); + candidate.setExperienceRaw(value(row, headerIndexes, "Experience")); + candidate.setCompanyDescription(value(row, headerIndexes, "AAB092")); + candidate.setIndustryType(value(row, headerIndexes, "IndustryType")); + candidate.setIndustrySub(value(row, headerIndexes, "IndustrySub")); + candidate.setCompanyNature(firstNonBlank(value(row, headerIndexes, "AAB019_OK"), value(row, headerIndexes, "AAB019"))); + candidate.setCompanyScale(firstNonBlank(value(row, headerIndexes, "Num_OK"), value(row, headerIndexes, "Num_employers"))); + candidate.setCollectDate(dateValue(row, headerIndexes, "Collect_time")); + String county = value(row, headerIndexes, "County"); + String city = value(row, headerIndexes, "City"); + candidate.setLocation(firstNonBlank(candidate.getJobAddress(), county, city)); + result.add(candidate); + } + return result; + } catch (IllegalArgumentException ex) { + throw ex; + } catch (Exception ex) { + throw new IOException("解析 Excel 失败", ex); + } + } + + private Map headers(Row row) { + Map result = new HashMap<>(); + for (Cell cell : row) { + String value = formatter.formatCellValue(cell); + if (value != null && !value.trim().isEmpty()) { + result.put(value.trim(), cell.getColumnIndex()); + } + } + return result; + } + + private String value(Row row, Map headers, String header) { + Integer column = headers.get(header); + if (column == null) { + return ""; + } + Cell cell = row.getCell(column); + return cell == null ? "" : formatter.formatCellValue(cell).trim(); + } + + private String dateValue(Row row, Map headers, String header) { + Integer column = headers.get(header); + if (column == null) { + return ""; + } + Cell cell = row.getCell(column); + if (cell == null) { + return ""; + } + if (DateUtil.isCellDateFormatted(cell)) { + Date value = cell.getDateCellValue(); + return value == null ? "" : dateFormat.format(value); + } + return formatter.formatCellValue(cell).trim(); + } + + private boolean isBlank(Row row) { + for (Cell cell : row) { + if (!formatter.formatCellValue(cell).trim().isEmpty()) { + return false; + } + } + return true; + } + + private String firstNonBlank(String... values) { + for (String value : values) { + if (value != null && !value.trim().isEmpty()) { + return value.trim(); + } + } + return ""; + } +} diff --git a/ruoyi-bussiness/src/main/java/com/ruoyi/cms/externalimport/ExternalJobFingerprint.java b/ruoyi-bussiness/src/main/java/com/ruoyi/cms/externalimport/ExternalJobFingerprint.java new file mode 100644 index 0000000..1d91d43 --- /dev/null +++ b/ruoyi-bussiness/src/main/java/com/ruoyi/cms/externalimport/ExternalJobFingerprint.java @@ -0,0 +1,107 @@ +package com.ruoyi.cms.externalimport; + +import java.net.URI; +import java.net.URISyntaxException; +import java.nio.charset.StandardCharsets; +import java.security.MessageDigest; +import java.security.NoSuchAlgorithmException; +import java.text.Normalizer; +import java.util.ArrayList; +import java.util.Collections; +import java.util.List; +import java.util.Locale; + +/** 岗位身份/内容指纹计算,保持纯函数以便单元测试。 */ +public final class ExternalJobFingerprint { + private static final String VERSION = "FP_V1"; + + private ExternalJobFingerprint() { + } + + public static String identity(ExternalJobCandidate candidate) { + String normalizedUrl = normalizeUrl(candidate.getJobUrl()); + if (!normalizedUrl.isEmpty()) { + return sha256(VERSION + "|" + normalizeText(candidate.getSourceCode()) + "|" + normalizedUrl); + } + return sha256(VERSION + "|" + normalizeText(candidate.getSourceCode()) + "|" + + normalizeText(candidate.getCompanyName()) + "|" + normalizeText(candidate.getJobTitle()) + "|" + + normalizeText(candidate.getLocation()) + "|" + normalizeText(candidate.getSalaryLow()) + "|" + + normalizeText(candidate.getSalaryHigh())); + } + + public static String content(ExternalJobCandidate candidate) { + return sha256(normalizeText(candidate.getJobTitle()) + "|" + normalizeText(candidate.getDescription()) + "|" + + normalizeText(candidate.getSalaryLow()) + "|" + normalizeText(candidate.getSalaryHigh()) + "|" + + normalizeText(candidate.getEducationCode()) + "|" + normalizeText(candidate.getExperienceCode()) + "|" + + normalizeText(candidate.getVacanciesText()) + "|" + normalizeText(candidate.getJobAddress()) + "|" + + normalizeText(candidate.getLocation()) + "|" + normalizeText(candidate.getAreaCode()) + "|" + + normalizeText(candidate.getJobCategory()) + "|" + normalizeText(candidate.getCompanyName()) + "|" + + normalizeUrl(candidate.getJobUrl()) + "|" + normalizeText(candidate.getPostingDate())); + } + + public static String normalizeText(String value) { + if (value == null) { + return ""; + } + return Normalizer.normalize(value, Normalizer.Form.NFKC) + .trim() + .replaceAll("\\s+", " ") + .toLowerCase(Locale.ROOT); + } + + public static String normalizeUrl(String value) { + if (value == null || value.trim().isEmpty()) { + return ""; + } + try { + URI uri = new URI(value.trim()); + String scheme = uri.getScheme() == null ? "https" : uri.getScheme().toLowerCase(Locale.ROOT); + String host = uri.getHost(); + if (host == null || host.trim().isEmpty()) { + return ""; + } + String path = uri.getRawPath() == null || uri.getRawPath().isEmpty() ? "/" : uri.getRawPath(); + if (path.length() > 1 && path.endsWith("/")) { + path = path.substring(0, path.length() - 1); + } + List queryParts = new ArrayList<>(); + if (uri.getRawQuery() != null && !uri.getRawQuery().isEmpty()) { + for (String part : uri.getRawQuery().split("&")) { + String key = part.split("=", 2)[0].toLowerCase(Locale.ROOT); + if (!key.startsWith("utm_") && !"spm".equals(key) && !"from".equals(key)) { + queryParts.add(part); + } + } + } + Collections.sort(queryParts); + return scheme + "://" + host.toLowerCase(Locale.ROOT) + path + + (queryParts.isEmpty() ? "" : "?" + join(queryParts, "&")); + } catch (URISyntaxException ex) { + return ""; + } + } + + private static String join(List values, String delimiter) { + StringBuilder builder = new StringBuilder(); + for (String value : values) { + if (builder.length() > 0) { + builder.append(delimiter); + } + builder.append(value); + } + return builder.toString(); + } + + private static String sha256(String value) { + try { + byte[] bytes = MessageDigest.getInstance("SHA-256").digest(value.getBytes(StandardCharsets.UTF_8)); + StringBuilder builder = new StringBuilder(64); + for (byte item : bytes) { + builder.append(String.format("%02x", item)); + } + return builder.toString(); + } catch (NoSuchAlgorithmException ex) { + throw new IllegalStateException("SHA-256 is unavailable", ex); + } + } +} diff --git a/ruoyi-bussiness/src/main/java/com/ruoyi/cms/externalimport/ExternalJobImportProperties.java b/ruoyi-bussiness/src/main/java/com/ruoyi/cms/externalimport/ExternalJobImportProperties.java new file mode 100644 index 0000000..3589118 --- /dev/null +++ b/ruoyi-bussiness/src/main/java/com/ruoyi/cms/externalimport/ExternalJobImportProperties.java @@ -0,0 +1,27 @@ +package com.ruoyi.cms.externalimport; + +import lombok.Data; +import org.springframework.boot.context.properties.ConfigurationProperties; +import org.springframework.stereotype.Component; + +import java.util.LinkedHashSet; +import java.util.Set; + +/** + * 外部岗位全量快照导入配置。 + * + * 令牌只能由环境专用 yml 或密钥管理系统提供,禁止提交真实值。 + */ +@Data +@Component +@ConfigurationProperties(prefix = "external-job-import") +public class ExternalJobImportProperties { + private boolean enabled = false; + private String latestMetadataUrl; + private String internalToken; + private int connectTimeoutMillis = 10000; + private int readTimeoutMillis = 120000; + private double maxInvalidRowRatio = 0.05D; + private double maxDeleteRatio = 0.30D; + private Set requiredSources = new LinkedHashSet<>(); +} diff --git a/ruoyi-bussiness/src/main/java/com/ruoyi/cms/externalimport/ExternalJobImportResult.java b/ruoyi-bussiness/src/main/java/com/ruoyi/cms/externalimport/ExternalJobImportResult.java new file mode 100644 index 0000000..a5a5094 --- /dev/null +++ b/ruoyi-bussiness/src/main/java/com/ruoyi/cms/externalimport/ExternalJobImportResult.java @@ -0,0 +1,20 @@ +package com.ruoyi.cms.externalimport; + +import lombok.Data; + +/** 一个成功或被拦截的导入批次摘要。 */ +@Data +public class ExternalJobImportResult { + private String batchId; + private String status; + private int totalRows; + private int validRows; + private int invalidRows; + private int duplicateRows; + private int insertCount; + private int updateCount; + private int keepCount; + private int restoreCount; + private int deleteCount; + private String message; +} diff --git a/ruoyi-bussiness/src/main/java/com/ruoyi/cms/externalimport/ExternalJobImportServiceImpl.java b/ruoyi-bussiness/src/main/java/com/ruoyi/cms/externalimport/ExternalJobImportServiceImpl.java new file mode 100644 index 0000000..5d418e2 --- /dev/null +++ b/ruoyi-bussiness/src/main/java/com/ruoyi/cms/externalimport/ExternalJobImportServiceImpl.java @@ -0,0 +1,667 @@ +package com.ruoyi.cms.externalimport; + +import com.fasterxml.jackson.core.JsonProcessingException; +import com.fasterxml.jackson.databind.ObjectMapper; +import org.springframework.jdbc.core.JdbcTemplate; +import org.springframework.stereotype.Service; +import org.springframework.transaction.annotation.Transactional; + +import java.io.IOException; +import java.io.InputStream; +import java.math.BigDecimal; +import java.sql.Timestamp; +import java.time.LocalDate; +import java.util.ArrayList; +import java.util.Collection; +import java.util.Collections; +import java.util.Comparator; +import java.util.HashMap; +import java.util.HashSet; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Locale; +import java.util.Map; +import java.util.Set; +import java.util.UUID; + +/** + * 外部岗位全量快照发布服务。 + * + * 本服务完全独立于旧的 JobServiceImpl.importRow:不清空 row_work,且只删除带外部标记的岗位。 + */ +@Service +public class ExternalJobImportServiceImpl implements IExternalJobImportService { + private static final String SYSTEM_USER = "external-import"; + + private final ExternalJobImportProperties properties; + private final ExternalUploadClient uploadClient; + private final JdbcTemplate jdbcTemplate; + private final ObjectMapper objectMapper; + private final ExternalJobExcelParser parser = new ExternalJobExcelParser(); + + public ExternalJobImportServiceImpl(ExternalJobImportProperties properties, + ExternalUploadClient uploadClient, + JdbcTemplate jdbcTemplate, + ObjectMapper objectMapper) { + this.properties = properties; + this.uploadClient = uploadClient; + this.jdbcTemplate = jdbcTemplate; + this.objectMapper = objectMapper; + } + + @Override + @Transactional(rollbackFor = Exception.class) + public ExternalJobImportResult importLatest() { + if (!properties.isEnabled()) { + throw new IllegalStateException("external-job-import.enabled 未开启"); + } + try { + ExternalUploadFileMetadata metadata = uploadClient.latest(); + if (hasSuccessfulSha256(metadata.getSha256())) { + ExternalJobImportResult result = new ExternalJobImportResult(); + result.setStatus("SKIPPED"); + result.setMessage("该 SHA-256 文件已经成功导入,已跳过"); + return result; + } + try (ExternalUploadClient.Download download = uploadClient.download(metadata)) { + return importStream(metadata, download.getInputStream()); + } + } catch (IOException ex) { + throw new IllegalStateException("拉取外部岗位 Excel 失败", ex); + } + } + + private ExternalJobImportResult importStream(ExternalUploadFileMetadata metadata, InputStream inputStream) throws IOException { + String batchId = UUID.randomUUID().toString().replace("-", ""); + List parsedRows = parser.parse(inputStream); + ExternalJobImportResult result = new ExternalJobImportResult(); + result.setBatchId(batchId); + result.setTotalRows(parsedRows.size()); + insertBatch(batchId, metadata, "VALIDATING", result); + + Map sourceCodeCache = new HashMap<>(); + Map dictionaryCodeCache = new HashMap<>(); + for (ExternalJobCandidate candidate : parsedRows) { + validateAndNormalize(candidate, sourceCodeCache, dictionaryCodeCache); + } + + int invalidRows = countInvalid(parsedRows); + result.setInvalidRows(invalidRows); + if (parsedRows.isEmpty()) { + blockBatch(batchId, result, "Excel 中没有有效数据行"); + return result; + } + if ((double) invalidRows / parsedRows.size() > properties.getMaxInvalidRowRatio()) { + persistRows(batchId, parsedRows, Collections.emptyMap(), Collections.emptyMap()); + blockBatch(batchId, result, "无效行比例超过阈值: " + properties.getMaxInvalidRowRatio()); + return result; + } + + Map current = deduplicate(parsedRows, result); + Set sourceScope = sources(current.values()); + if (current.isEmpty()) { + persistRows(batchId, parsedRows, Collections.emptyMap(), current); + blockBatch(batchId, result, "没有可发布的有效岗位"); + return result; + } + if (!sourceScope.containsAll(properties.getRequiredSources())) { + persistRows(batchId, parsedRows, Collections.emptyMap(), current); + blockBatch(batchId, result, "本批次缺少必需来源: " + missingSources(sourceScope)); + return result; + } + + Map previous = activeExternalJobs(sourceScope); + int prospectiveDeletes = previous.size() - intersectionSize(previous.keySet(), current.keySet()); + if (!previous.isEmpty() && (double) prospectiveDeletes / previous.size() > properties.getMaxDeleteRatio()) { + persistRows(batchId, parsedRows, Collections.emptyMap(), current); + blockBatch(batchId, result, "拟删除比例 " + prospectiveDeletes + "/" + previous.size() + " 超过阈值: " + properties.getMaxDeleteRatio()); + return result; + } + + Map fingerprintToJobId = new HashMap<>(); + for (ExternalJobCandidate candidate : current.values()) { + ExistingJob existing = previous.remove(candidate.getJobFingerprint()); + Long companyId = resolveCompany(candidate); + if (existing != null) { + if (candidate.getContentFingerprint().equals(existing.contentFingerprint)) { + touchExistingJob(existing.jobId, batchId); + result.setKeepCount(result.getKeepCount() + 1); + } else { + updateExistingJob(existing.jobId, candidate, companyId, batchId); + result.setUpdateCount(result.getUpdateCount() + 1); + } + fingerprintToJobId.put(candidate.getJobFingerprint(), existing.jobId); + continue; + } + Long deletedJobId = deletedExternalJob(candidate.getJobFingerprint()); + if (deletedJobId != null) { + restoreJob(deletedJobId, candidate, companyId, batchId); + result.setRestoreCount(result.getRestoreCount() + 1); + fingerprintToJobId.put(candidate.getJobFingerprint(), deletedJobId); + } else { + Long jobId = insertJob(candidate, companyId, batchId); + result.setInsertCount(result.getInsertCount() + 1); + fingerprintToJobId.put(candidate.getJobFingerprint(), jobId); + } + } + + for (ExistingJob absent : previous.values()) { + logicallyDeleteJob(absent.jobId, batchId); + result.setDeleteCount(result.getDeleteCount() + 1); + } + persistRows(batchId, parsedRows, fingerprintToJobId, current); + recordMonitoring(batchId, parsedRows, current.values(), result); + result.setValidRows(current.size()); + result.setStatus("SUCCESS"); + result.setMessage("外部岗位全量快照发布成功"); + updateBatch(batchId, "SUCCESS", result, null); + return result; + } + + @Override + public List> listBatches(String status, Integer limit) { + int safeLimit = limit == null ? 50 : Math.max(1, Math.min(limit, 200)); + if (isBlank(status)) { + return jdbcTemplate.queryForList("SELECT * FROM job_import_batch ORDER BY created_at DESC LIMIT ?", safeLimit); + } + return jdbcTemplate.queryForList("SELECT * FROM job_import_batch WHERE status = ? ORDER BY created_at DESC LIMIT ?", status, safeLimit); + } + + @Override + public List> listSourceAliases() { + return jdbcTemplate.queryForList("SELECT alias.*, website.website_name, website.website_url " + + "FROM job_import_source_alias alias " + + "LEFT JOIN website_management website ON website.website_id = alias.website_id " + + "ORDER BY alias.is_active DESC, alias.alias_name"); + } + + @Override + public void saveSourceAlias(ExternalJobSourceAliasRequest request) { + if (request == null || isBlank(request.getAliasName()) || isBlank(request.getSourceCode())) { + throw new IllegalArgumentException("aliasName 和 sourceCode 为必填项"); + } + String active = "0".equals(request.getIsActive()) ? "0" : "1"; + String snapshotRequired = "0".equals(request.getSnapshotRequired()) ? "0" : "1"; + jdbcTemplate.update("INSERT INTO job_import_source_alias (alias_name, source_code, website_id, is_active, snapshot_required, remark, create_time, update_time) " + + "VALUES (?, ?, ?, ?, ?, ?, CURRENT_TIMESTAMP, CURRENT_TIMESTAMP) " + + "ON CONFLICT (alias_name) DO UPDATE SET source_code=EXCLUDED.source_code, website_id=EXCLUDED.website_id, " + + "is_active=EXCLUDED.is_active, snapshot_required=EXCLUDED.snapshot_required, remark=EXCLUDED.remark, update_time=CURRENT_TIMESTAMP", + request.getAliasName().trim(), request.getSourceCode().trim(), request.getWebsiteId(), active, snapshotRequired, emptyToNull(request.getRemark())); + } + + private void validateAndNormalize(ExternalJobCandidate candidate, Map sourceCodeCache, + Map dictionaryCodeCache) { + candidate.setJobTitle(trim(candidate.getJobTitle())); + candidate.setCompanyName(trim(candidate.getCompanyName())); + candidate.setDescription(trim(candidate.getDescription())); + candidate.setJobUrl(trim(candidate.getJobUrl())); + candidate.setSourceName(trim(candidate.getSourceName())); + if (isBlank(candidate.getJobTitle())) { + candidate.getErrors().add("岗位名称 Aca112 为空"); + } + if (isBlank(candidate.getCompanyName())) { + candidate.getErrors().add("企业名称 AAB004 为空"); + } + if (isBlank(candidate.getDescription())) { + candidate.getErrors().add("岗位描述 Acb22a 为空"); + } + if (isBlank(candidate.getSourceName())) { + candidate.getErrors().add("来源 ORG 为空"); + } else { + String sourceCode = sourceCodeCache.get(candidate.getSourceName()); + if (sourceCode == null) { + List values = jdbcTemplate.queryForList( + "SELECT source_code FROM job_import_source_alias WHERE alias_name = ? AND is_active = '1'", String.class, candidate.getSourceName()); + sourceCode = values.size() == 1 ? values.get(0) : ""; + sourceCodeCache.put(candidate.getSourceName(), sourceCode); + } + if (isBlank(sourceCode)) { + candidate.getErrors().add("来源未在 job_import_source_alias 中启用: " + candidate.getSourceName()); + } else { + candidate.setSourceCode(sourceCode); + } + } + if (isBlank(candidate.getJobUrl()) || isBlank(ExternalJobFingerprint.normalizeUrl(candidate.getJobUrl()))) { + candidate.getErrors().add("岗位链接 ACE760 为空或格式无效"); + } + Long low = parseMoney(candidate.getSalaryLow()); + Long high = parseMoney(candidate.getSalaryHigh()); + if (low == null || high == null) { + candidate.getErrors().add("薪资字段必须为数字"); + } else if (low > high) { + candidate.getErrors().add("最低薪资不能大于最高薪资"); + } + if (!isDate(candidate.getPostingDate())) { + candidate.getErrors().add("发布日期 Aae397 无法解析"); + } + if (!isDate(candidate.getCollectDate())) { + candidate.getErrors().add("采集日期 Collect_time 无法解析"); + } + candidate.setEducationCode(educationCode(candidate.getEducationRaw(), candidate)); + candidate.setExperienceCode(experienceCode(candidate.getExperienceRaw(), candidate)); + candidate.setAreaCode(areaCode(candidate.getLocation(), candidate)); + candidate.setCompanyScaleCode(dictionaryCode("scale", candidate.getCompanyScale(), candidate, dictionaryCodeCache)); + candidate.setCompanyNatureCode(dictionaryCode("company_nature", candidate.getCompanyNature(), candidate, dictionaryCodeCache)); + if (isBlank(candidate.getJobAddress())) { + candidate.getWarnings().add("岗位详细地址为空,已使用区县或城市作为岗位地点"); + } + if (candidate.isValid()) { + candidate.setJobFingerprint(ExternalJobFingerprint.identity(candidate)); + candidate.setContentFingerprint(ExternalJobFingerprint.content(candidate)); + } + } + + private Map deduplicate(List candidates, ExternalJobImportResult result) { + Map current = new LinkedHashMap<>(); + for (ExternalJobCandidate candidate : candidates) { + if (!candidate.isValid()) { + continue; + } + ExternalJobCandidate before = current.get(candidate.getJobFingerprint()); + if (before == null) { + current.put(candidate.getJobFingerprint(), candidate); + } else { + result.setDuplicateRows(result.getDuplicateRows() + 1); + before.getWarnings().add("批内重复指纹,保留信息更完整的一行"); + if (isBetter(candidate, before)) { + current.put(candidate.getJobFingerprint(), candidate); + } + } + } + return current; + } + + private boolean isBetter(ExternalJobCandidate left, ExternalJobCandidate right) { + int completeness = Integer.compare(left.completenessScore(), right.completenessScore()); + if (completeness != 0) { + return completeness > 0; + } + return trim(left.getCollectDate()).compareTo(trim(right.getCollectDate())) > 0; + } + + private Map activeExternalJobs(Set sourceCodes) { + if (sourceCodes.isEmpty()) { + return Collections.emptyMap(); + } + String placeholders = placeholders(sourceCodes.size()); + List args = new ArrayList(sourceCodes); + List jobs = jdbcTemplate.query( + "SELECT job_id, job_fingerprint, job_content_fingerprint FROM job " + + "WHERE external_import_flag = '1' AND del_flag = '0' " + + "AND external_source_code IN (" + placeholders + ")", + (rs, rowNum) -> new ExistingJob(rs.getLong("job_id"), rs.getString("job_fingerprint"), rs.getString("job_content_fingerprint")), + args.toArray()); + Map result = new HashMap<>(); + for (ExistingJob job : jobs) { + result.put(job.fingerprint, job); + } + return result; + } + + private Long deletedExternalJob(String fingerprint) { + List ids = jdbcTemplate.queryForList( + "SELECT job_id FROM job WHERE external_import_flag = '1' AND del_flag = '2' " + + "AND job_fingerprint = ? ORDER BY update_time DESC LIMIT 1", Long.class, fingerprint); + return ids.isEmpty() ? null : ids.get(0); + } + + private Long resolveCompany(ExternalJobCandidate candidate) { + List ids = jdbcTemplate.queryForList( + "SELECT company_id FROM company WHERE del_flag = '0' AND LOWER(BTRIM(name)) = LOWER(BTRIM(?))", Long.class, candidate.getCompanyName()); + if (ids.size() > 1) { + candidate.getWarnings().add("匹配到多个同名企业,岗位未绑定 company_id"); + return null; + } + if (ids.size() == 1) { + Long companyId = ids.get(0); + jdbcTemplate.update("UPDATE company SET " + + "location = CASE WHEN NULLIF(BTRIM(location), '') IS NULL THEN ? ELSE location END, " + + "description = CASE WHEN NULLIF(BTRIM(description), '') IS NULL THEN ? ELSE description END, " + + "industry = CASE WHEN NULLIF(BTRIM(industry), '') IS NULL THEN ? ELSE industry END, " + + "scale = CASE WHEN NULLIF(BTRIM(scale), '') IS NULL THEN ? ELSE scale END, " + + "company_nature = CASE WHEN NULLIF(BTRIM(company_nature), '') IS NULL THEN ? ELSE company_nature END, " + + "contact_person = CASE WHEN NULLIF(BTRIM(contact_person), '') IS NULL THEN ? ELSE contact_person END, " + + "update_by = ?, update_time = CURRENT_TIMESTAMP WHERE company_id = ?", + emptyToNull(candidate.getLocation()), emptyToNull(candidate.getCompanyDescription()), emptyToNull(firstNonBlank(candidate.getIndustrySub(), candidate.getIndustryType())), + emptyToNull(candidate.getCompanyScaleCode()), emptyToNull(candidate.getCompanyNatureCode()), emptyToNull(candidate.getContactPerson()), SYSTEM_USER, companyId); + return companyId; + } + Long companyId = jdbcTemplate.queryForObject( + "INSERT INTO company (name, location, industry, scale, company_nature, description, contact_person, del_flag, create_by, create_time, company_status) " + + "VALUES (?, ?, ?, ?, ?, ?, ?, '0', ?, CURRENT_TIMESTAMP, '0') RETURNING company_id", + Long.class, candidate.getCompanyName(), emptyToNull(candidate.getLocation()), emptyToNull(firstNonBlank(candidate.getIndustrySub(), candidate.getIndustryType())), + emptyToNull(candidate.getCompanyScaleCode()), emptyToNull(candidate.getCompanyNatureCode()), emptyToNull(candidate.getCompanyDescription()), emptyToNull(candidate.getContactPerson()), SYSTEM_USER); + return companyId; + } + + private Long insertJob(ExternalJobCandidate candidate, Long companyId, String batchId) { + return jdbcTemplate.queryForObject("INSERT INTO job (job_title, min_salary, max_salary, education, experience, company_name, " + + "job_location, job_location_area_code, posting_date, vacancies, \"view\", company_id, is_hot, is_urgent, apply_num, " + + "description, is_publish, data_source, job_url, job_category, job_address, job_status, del_flag, create_by, create_time, " + + "external_import_flag, external_source_code, external_source_name, job_fingerprint, job_content_fingerprint, " + + "external_batch_id, external_first_seen_at, external_last_seen_at) " + + "VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, 0, ?, '0', '0', '0', ?, 1, ?, ?, ?, ?, '0', '0', ?, CURRENT_TIMESTAMP, " + + "'1', ?, ?, ?, ?, ?, CURRENT_TIMESTAMP, CURRENT_TIMESTAMP) RETURNING job_id", + Long.class, + candidate.getJobTitle(), parseMoney(candidate.getSalaryLow()), parseMoney(candidate.getSalaryHigh()), candidate.getEducationCode(), candidate.getExperienceCode(), + candidate.getCompanyName(), emptyToNull(candidate.getLocation()), emptyToNull(candidate.getAreaCode()), timestamp(candidate.getPostingDate()), vacancies(candidate.getVacanciesText()), + companyId, candidate.getDescription(), candidate.getSourceName(), candidate.getJobUrl(), emptyToNull(candidate.getJobCategory()), emptyToNull(candidate.getJobAddress()), + SYSTEM_USER, candidate.getSourceCode(), candidate.getSourceName(), candidate.getJobFingerprint(), candidate.getContentFingerprint(), batchId); + } + + private void updateExistingJob(Long jobId, ExternalJobCandidate candidate, Long companyId, String batchId) { + jdbcTemplate.update("UPDATE job SET job_title=?, min_salary=?, max_salary=?, education=?, experience=?, company_name=?, " + + "job_location=?, job_location_area_code=?, posting_date=?, vacancies=?, company_id=?, description=?, is_publish=1, " + + "data_source=?, job_url=?, job_category=?, job_address=?, job_status='0', del_flag='0', update_by=?, update_time=CURRENT_TIMESTAMP, " + + "external_source_code=?, external_source_name=?, job_content_fingerprint=?, external_batch_id=?, external_last_seen_at=CURRENT_TIMESTAMP " + + "WHERE job_id=?", + candidate.getJobTitle(), parseMoney(candidate.getSalaryLow()), parseMoney(candidate.getSalaryHigh()), candidate.getEducationCode(), candidate.getExperienceCode(), + candidate.getCompanyName(), emptyToNull(candidate.getLocation()), emptyToNull(candidate.getAreaCode()), timestamp(candidate.getPostingDate()), vacancies(candidate.getVacanciesText()), + companyId, candidate.getDescription(), candidate.getSourceName(), candidate.getJobUrl(), emptyToNull(candidate.getJobCategory()), emptyToNull(candidate.getJobAddress()), + SYSTEM_USER, candidate.getSourceCode(), candidate.getSourceName(), candidate.getContentFingerprint(), batchId, jobId); + } + + private void restoreJob(Long jobId, ExternalJobCandidate candidate, Long companyId, String batchId) { + updateExistingJob(jobId, candidate, companyId, batchId); + } + + private void touchExistingJob(Long jobId, String batchId) { + jdbcTemplate.update("UPDATE job SET external_batch_id=?, external_last_seen_at=CURRENT_TIMESTAMP, update_by=?, update_time=CURRENT_TIMESTAMP WHERE job_id=?", + batchId, SYSTEM_USER, jobId); + } + + private void logicallyDeleteJob(Long jobId, String batchId) { + jdbcTemplate.update("UPDATE job SET del_flag='2', job_status='1', is_publish=0, external_batch_id=?, update_by=?, update_time=CURRENT_TIMESTAMP WHERE job_id=?", + batchId, SYSTEM_USER, jobId); + } + + private void insertBatch(String batchId, ExternalUploadFileMetadata metadata, String status, ExternalJobImportResult result) { + jdbcTemplate.update("INSERT INTO job_import_batch (batch_id, file_id, file_name, file_sha256, snapshot_scope, fingerprint_version, status, total_rows, " + + "valid_rows, invalid_rows, duplicate_rows, insert_count, update_count, keep_count, restore_count, delete_count, created_at, updated_at) " + + "VALUES (?, ?, ?, ?, ?, 'FP_V1', ?, ?, 0, 0, 0, 0, 0, 0, 0, 0, CURRENT_TIMESTAMP, CURRENT_TIMESTAMP)", + batchId, metadata.getFileId(), metadata.getFilename(), metadata.getSha256(), join(metadata.getSourceScope()), status, result.getTotalRows()); + } + + private void updateBatch(String batchId, String status, ExternalJobImportResult result, String error) { + jdbcTemplate.update("UPDATE job_import_batch SET status=?, valid_rows=?, invalid_rows=?, duplicate_rows=?, insert_count=?, update_count=?, keep_count=?, " + + "restore_count=?, delete_count=?, error_summary=?, completed_at=CASE WHEN ? IN ('SUCCESS','FAILED','BLOCKED') THEN CURRENT_TIMESTAMP ELSE completed_at END, " + + "updated_at=CURRENT_TIMESTAMP WHERE batch_id=?", + status, result.getValidRows(), result.getInvalidRows(), result.getDuplicateRows(), result.getInsertCount(), result.getUpdateCount(), result.getKeepCount(), + result.getRestoreCount(), result.getDeleteCount(), error, status, batchId); + } + + private void blockBatch(String batchId, ExternalJobImportResult result, String message) { + result.setStatus("BLOCKED"); + result.setMessage(message); + updateBatch(batchId, "BLOCKED", result, message); + } + + private void persistRows(String batchId, List candidates, Map fingerprintToJobId, + Map publishedCandidates) { + Set retained = fingerprintToJobId.keySet(); + for (ExternalJobCandidate candidate : candidates) { + boolean canonicalRow = candidate.isValid() && publishedCandidates.get(candidate.getJobFingerprint()) == candidate; + String validationStatus = candidate.isValid() ? (canonicalRow ? "VALID" : "DUPLICATE") : "INVALID"; + String resultStatus = canonicalRow && retained.contains(candidate.getJobFingerprint()) ? "PUBLISHED" : candidate.isValid() ? "SKIPPED" : "FAILED"; + Long jobId = canonicalRow ? fingerprintToJobId.get(candidate.getJobFingerprint()) : null; + jdbcTemplate.update("INSERT INTO job_import_row (batch_id, excel_row_number, source_name, source_code, job_fingerprint, content_fingerprint, " + + "validation_status, result_status, error_message, raw_payload, job_id, created_at) " + + "VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, CURRENT_TIMESTAMP)", + batchId, candidate.getExcelRowNumber(), candidate.getSourceName(), candidate.getSourceCode(), candidate.getJobFingerprint(), candidate.getContentFingerprint(), + validationStatus, resultStatus, join(candidate.getErrors()), rawPayload(candidate), jobId); + } + } + + /** 将批次结果同步到现有的入库监测与采集趋势表,避免后台手工补录。 */ + private void recordMonitoring(String batchId, List parsedRows, + Collection publishedRows, + ExternalJobImportResult result) { + String summary = "新增" + result.getInsertCount() + "个,更新" + result.getUpdateCount() + + "个,保留" + result.getKeepCount() + "个,恢复" + result.getRestoreCount() + + "个,下线" + result.getDeleteCount() + "个,失败" + result.getInvalidRows() + "个"; + jdbcTemplate.update("INSERT INTO job_data_storage_detection (detection_id, storage_date, storage_number, storage_result, failed_reason, del_flag) " + + "VALUES (?, CURRENT_TIMESTAMP, ?, ?, ?, '0')", + batchId, publishedRows.size(), summary, result.getInvalidRows() == 0 ? null : "存在字段校验失败行,请查看 job_import_row"); + jdbcTemplate.update("INSERT INTO job_data_trend (delete_count, insert_count, fail_count, storage_time, website_name, del_flag) " + + "VALUES (?, ?, ?, CURRENT_TIMESTAMP, 'total', '0')", + result.getDeleteCount(), result.getInsertCount() + result.getRestoreCount(), result.getInvalidRows() + result.getDuplicateRows()); + + Map successBySource = new HashMap<>(); + Map failedBySource = new HashMap<>(); + Map sourceNames = new HashMap<>(); + for (ExternalJobCandidate candidate : publishedRows) { + increment(successBySource, candidate.getSourceCode()); + sourceNames.put(candidate.getSourceCode(), candidate.getSourceName()); + } + for (ExternalJobCandidate candidate : parsedRows) { + if (!candidate.isValid()) { + increment(failedBySource, candidate.getSourceCode()); + if (!isBlank(candidate.getSourceCode())) { + sourceNames.put(candidate.getSourceCode(), candidate.getSourceName()); + } + } + } + Set sources = new HashSet<>(); + sources.addAll(successBySource.keySet()); + sources.addAll(failedBySource.keySet()); + for (String sourceCode : sources) { + if (isBlank(sourceCode)) { + continue; + } + List websiteIds = jdbcTemplate.queryForList( + "SELECT website_id FROM job_import_source_alias WHERE source_code=? AND is_active='1' LIMIT 1", Long.class, sourceCode); + Long websiteId = websiteIds.isEmpty() ? null : websiteIds.get(0); + int success = value(successBySource, sourceCode); + int failed = value(failedBySource, sourceCode); + jdbcTemplate.update("INSERT INTO job_data_storage_detail (detection_id, website_id, website_name, storage_time, success_number, failed_number, storage_detail, failed_reason) " + + "VALUES (?, ?, ?, CURRENT_TIMESTAMP, ?, ?, ?, ?)", + batchId, websiteId, sourceNames.get(sourceCode), success, failed, + "外部导入批次 " + batchId, failed == 0 ? null : "存在字段校验失败行"); + jdbcTemplate.update("INSERT INTO job_data_trend (delete_count, insert_count, fail_count, storage_time, website_name, del_flag) " + + "VALUES (0, ?, ?, CURRENT_TIMESTAMP, ?, '0')", success, failed, sourceCode); + } + } + + private String rawPayload(ExternalJobCandidate candidate) { + try { + return objectMapper.writeValueAsString(candidate); + } catch (JsonProcessingException ex) { + return "{\"excelRowNumber\":" + candidate.getExcelRowNumber() + "}"; + } + } + + private void increment(Map values, String key) { + if (!isBlank(key)) { + values.put(key, value(values, key) + 1); + } + } + + private int value(Map values, String key) { + Integer value = values.get(key); + return value == null ? 0 : value; + } + + private boolean hasSuccessfulSha256(String sha256) { + Integer count = jdbcTemplate.queryForObject("SELECT COUNT(1) FROM job_import_batch WHERE file_sha256=? AND status='SUCCESS'", Integer.class, sha256); + return count != null && count > 0; + } + + private int countInvalid(Collection candidates) { + int result = 0; + for (ExternalJobCandidate candidate : candidates) { + if (!candidate.isValid()) { + result++; + } + } + return result; + } + + private Set sources(Collection candidates) { + Set result = new HashSet<>(); + for (ExternalJobCandidate candidate : candidates) { + result.add(candidate.getSourceCode()); + } + return result; + } + + private int intersectionSize(Set first, Set second) { + int count = 0; + for (String value : first) { + if (second.contains(value)) { + count++; + } + } + return count; + } + + private String missingSources(Set scope) { + Set missing = new HashSet<>(properties.getRequiredSources()); + missing.removeAll(scope); + return join(missing); + } + + private String educationCode(String raw, ExternalJobCandidate candidate) { + String value = trim(raw); + if ("不限".equals(value) || "学历不限".equals(value)) return "-1"; + if ("初中及以下".equals(value) || "高中以下".equals(value)) return "0"; + if ("中专/中技".equals(value) || "中专/技校".equals(value) || "中技/中专".equals(value)) return "1"; + if ("高中".equals(value)) return "2"; + if ("大专".equals(value)) return "3"; + if ("本科".equals(value) || "统招本科".equals(value)) return "4"; + if ("硕士".equals(value)) return "5"; + if ("博士".equals(value)) return "6"; + candidate.getWarnings().add("学历未映射,按不限处理: " + value); + return "-1"; + } + + private String experienceCode(String raw, ExternalJobCandidate candidate) { + String value = trim(raw); + if ("不限".equals(value) || "经验不限".equals(value)) return "0"; + if ("实习生".equals(value)) return "1"; + if ("应届毕业生".equals(value)) return "2"; + if ("1年以下".equals(value) || "1年".equals(value)) return "3"; + if ("1-3年".equals(value) || "1-3年工作经验".equals(value) || "2年".equals(value) || "3年".equals(value)) return "4"; + if ("3-5年".equals(value)) return "5"; + if ("5-10年".equals(value)) return "6"; + if ("10年以上".equals(value)) return "7"; + candidate.getWarnings().add("经验未映射,按经验不限处理: " + value); + return "0"; + } + + private String areaCode(String location, ExternalJobCandidate candidate) { + if (isBlank(location)) { + return null; + } + List values = jdbcTemplate.queryForList("SELECT dict_value FROM bussiness_dict_data WHERE dict_type='area' AND dict_label=? AND status='0' LIMIT 1", String.class, location); + if (values.isEmpty()) { + candidate.getWarnings().add("未匹配到区域字典: " + location); + return null; + } + return values.get(0); + } + + private String dictionaryCode(String type, String raw, ExternalJobCandidate candidate, Map cache) { + if (isBlank(raw)) { + return null; + } + String label = trim(raw); + String cacheKey = type + "|" + label; + if (cache.containsKey(cacheKey)) { + return emptyToNull(cache.get(cacheKey)); + } + List values = jdbcTemplate.queryForList( + "SELECT dict_value FROM bussiness_dict_data WHERE dict_type=? AND dict_label=? AND status='0' LIMIT 1", + String.class, type, label); + if (values.isEmpty() && "scale".equals(type) && !label.endsWith("人")) { + values = jdbcTemplate.queryForList( + "SELECT dict_value FROM bussiness_dict_data WHERE dict_type=? AND dict_label=? AND status='0' LIMIT 1", + String.class, type, label + "人"); + } + String code = values.isEmpty() ? "" : values.get(0); + cache.put(cacheKey, code); + if (isBlank(code)) { + candidate.getWarnings().add("未匹配到 " + type + " 字典: " + label); + return null; + } + return code; + } + + private Long parseMoney(String value) { + try { + if (isBlank(value)) return null; + return new BigDecimal(trim(value)).longValueExact(); + } catch (Exception ex) { + return null; + } + } + + private Long vacancies(String value) { + if (isBlank(value)) return -1L; + String normalized = value.replace("人", "").replace("以上", "").trim(); + if (normalized.contains("-")) { + normalized = normalized.substring(normalized.lastIndexOf('-') + 1).trim(); + } + try { + return new BigDecimal(normalized.replaceAll("[^0-9.]", "")).longValue(); + } catch (Exception ex) { + return -1L; + } + } + + private Timestamp timestamp(String value) { + return Timestamp.valueOf(LocalDate.parse(value).atStartOfDay()); + } + + private boolean isDate(String value) { + try { + LocalDate.parse(value); + return true; + } catch (Exception ex) { + return false; + } + } + + private String placeholders(int size) { + List values = new ArrayList<>(); + for (int i = 0; i < size; i++) values.add("?"); + return join(values); + } + + private String trim(String value) { + return value == null ? "" : value.trim(); + } + + private String firstNonBlank(String first, String second) { + return !isBlank(first) ? first : second; + } + + private String emptyToNull(String value) { + return isBlank(value) ? null : value.trim(); + } + + private boolean isBlank(String value) { + return value == null || value.trim().isEmpty(); + } + + private String join(Collection values) { + StringBuilder builder = new StringBuilder(); + for (String value : values) { + if (isBlank(value)) continue; + if (builder.length() > 0) builder.append(","); + builder.append(value); + } + return builder.toString(); + } + + private static final class ExistingJob { + private final Long jobId; + private final String fingerprint; + private final String contentFingerprint; + + private ExistingJob(Long jobId, String fingerprint, String contentFingerprint) { + this.jobId = jobId; + this.fingerprint = fingerprint; + this.contentFingerprint = contentFingerprint; + } + } +} diff --git a/ruoyi-bussiness/src/main/java/com/ruoyi/cms/externalimport/ExternalJobSourceAliasRequest.java b/ruoyi-bussiness/src/main/java/com/ruoyi/cms/externalimport/ExternalJobSourceAliasRequest.java new file mode 100644 index 0000000..1f2ddca --- /dev/null +++ b/ruoyi-bussiness/src/main/java/com/ruoyi/cms/externalimport/ExternalJobSourceAliasRequest.java @@ -0,0 +1,14 @@ +package com.ruoyi.cms.externalimport; + +import lombok.Data; + +/** 外部 Excel ORG 到系统稳定来源编码的后台维护请求。 */ +@Data +public class ExternalJobSourceAliasRequest { + private String aliasName; + private String sourceCode; + private Long websiteId; + private String isActive = "1"; + private String snapshotRequired = "1"; + private String remark; +} diff --git a/ruoyi-bussiness/src/main/java/com/ruoyi/cms/externalimport/ExternalUploadClient.java b/ruoyi-bussiness/src/main/java/com/ruoyi/cms/externalimport/ExternalUploadClient.java new file mode 100644 index 0000000..d14783a --- /dev/null +++ b/ruoyi-bussiness/src/main/java/com/ruoyi/cms/externalimport/ExternalUploadClient.java @@ -0,0 +1,102 @@ +package com.ruoyi.cms.externalimport; + +import com.fasterxml.jackson.databind.JsonNode; +import com.fasterxml.jackson.databind.ObjectMapper; +import org.springframework.stereotype.Component; + +import java.io.BufferedInputStream; +import java.io.IOException; +import java.io.InputStream; +import java.net.HttpURLConnection; +import java.net.URL; + +/** 与外部 Excel 上传系统交互的最小内部客户端。 */ +@Component +public class ExternalUploadClient { + private final ExternalJobImportProperties properties; + private final ObjectMapper objectMapper; + + public ExternalUploadClient(ExternalJobImportProperties properties, ObjectMapper objectMapper) { + this.properties = properties; + this.objectMapper = objectMapper; + } + + public ExternalUploadFileMetadata latest() throws IOException { + if (isBlank(properties.getLatestMetadataUrl())) { + throw new IllegalStateException("external-job-import.latest-metadata-url 未配置"); + } + HttpURLConnection connection = open(properties.getLatestMetadataUrl()); + try { + requireOk(connection, "获取最新外部 Excel 元数据失败"); + JsonNode root = objectMapper.readTree(connection.getInputStream()); + JsonNode data = root.path("data"); + if (data.isMissingNode() || data.isNull()) { + throw new IOException("外部上传服务没有返回可导入文件"); + } + ExternalUploadFileMetadata metadata = objectMapper.treeToValue(data, ExternalUploadFileMetadata.class); + if (isBlank(metadata.getFileId()) || isBlank(metadata.getSha256()) || isBlank(metadata.getDownloadUrl())) { + throw new IOException("外部上传服务返回的文件元数据不完整"); + } + metadata.setDownloadUrl(resolve(properties.getLatestMetadataUrl(), metadata.getDownloadUrl())); + return metadata; + } finally { + connection.disconnect(); + } + } + + public Download download(ExternalUploadFileMetadata metadata) throws IOException { + HttpURLConnection connection = open(metadata.getDownloadUrl()); + requireOk(connection, "下载外部 Excel 文件失败"); + return new Download(connection, new BufferedInputStream(connection.getInputStream())); + } + + private HttpURLConnection open(String target) throws IOException { + HttpURLConnection connection = (HttpURLConnection) new URL(target).openConnection(); + connection.setRequestMethod("GET"); + connection.setConnectTimeout(properties.getConnectTimeoutMillis()); + connection.setReadTimeout(properties.getReadTimeoutMillis()); + connection.setRequestProperty("Accept", "application/json, application/vnd.openxmlformats-officedocument.spreadsheetml.sheet"); + if (!isBlank(properties.getInternalToken())) { + connection.setRequestProperty("X-Internal-Token", properties.getInternalToken()); + } + return connection; + } + + private void requireOk(HttpURLConnection connection, String message) throws IOException { + int status = connection.getResponseCode(); + if (status != HttpURLConnection.HTTP_OK) { + throw new IOException(message + ",HTTP 状态码: " + status); + } + } + + private String resolve(String base, String value) throws IOException { + return new URL(new URL(base), value).toExternalForm(); + } + + private boolean isBlank(String value) { + return value == null || value.trim().isEmpty(); + } + + public static final class Download implements AutoCloseable { + private final HttpURLConnection connection; + private final InputStream inputStream; + + private Download(HttpURLConnection connection, InputStream inputStream) { + this.connection = connection; + this.inputStream = inputStream; + } + + public InputStream getInputStream() { + return inputStream; + } + + @Override + public void close() throws IOException { + try { + inputStream.close(); + } finally { + connection.disconnect(); + } + } + } +} diff --git a/ruoyi-bussiness/src/main/java/com/ruoyi/cms/externalimport/ExternalUploadFileMetadata.java b/ruoyi-bussiness/src/main/java/com/ruoyi/cms/externalimport/ExternalUploadFileMetadata.java new file mode 100644 index 0000000..e5d198a --- /dev/null +++ b/ruoyi-bussiness/src/main/java/com/ruoyi/cms/externalimport/ExternalUploadFileMetadata.java @@ -0,0 +1,22 @@ +package com.ruoyi.cms.externalimport; + +import com.fasterxml.jackson.annotation.JsonProperty; +import lombok.Data; + +import java.util.ArrayList; +import java.util.List; + +/** 外部上传服务返回的一个可导入文件版本。 */ +@Data +public class ExternalUploadFileMetadata { + @JsonProperty("file_id") + private String fileId; + private String filename; + private String sha256; + @JsonProperty("download_url") + private String downloadUrl; + @JsonProperty("upload_time") + private String uploadTime; + @JsonProperty("source_scope") + private List sourceScope = new ArrayList<>(); +} diff --git a/ruoyi-bussiness/src/main/java/com/ruoyi/cms/externalimport/IExternalJobImportService.java b/ruoyi-bussiness/src/main/java/com/ruoyi/cms/externalimport/IExternalJobImportService.java new file mode 100644 index 0000000..30ea9f7 --- /dev/null +++ b/ruoyi-bussiness/src/main/java/com/ruoyi/cms/externalimport/IExternalJobImportService.java @@ -0,0 +1,14 @@ +package com.ruoyi.cms.externalimport; + +import java.util.List; +import java.util.Map; + +public interface IExternalJobImportService { + ExternalJobImportResult importLatest(); + + List> listBatches(String status, Integer limit); + + List> listSourceAliases(); + + void saveSourceAlias(ExternalJobSourceAliasRequest request); +} diff --git a/ruoyi-bussiness/src/main/java/com/ruoyi/cms/service/impl/JobDataTrendServiceImpl.java b/ruoyi-bussiness/src/main/java/com/ruoyi/cms/service/impl/JobDataTrendServiceImpl.java index e7238c3..8818360 100644 --- a/ruoyi-bussiness/src/main/java/com/ruoyi/cms/service/impl/JobDataTrendServiceImpl.java +++ b/ruoyi-bussiness/src/main/java/com/ruoyi/cms/service/impl/JobDataTrendServiceImpl.java @@ -85,7 +85,9 @@ public class JobDataTrendServiceImpl extends ServiceImpl lqw = Wrappers.lambdaQuery(); - lqw.eq(JobDataTrend::getWebsiteName, "total"); + String sourceCode = params != null && StringUtils.isNotEmpty(params.get("sourceCode")) + ? params.get("sourceCode") : "total"; + lqw.eq(JobDataTrend::getWebsiteName, sourceCode); // 查询范围: >= Start 00:00:00 且 <= End 23:59:59 // 注意:数据库存的是 Date (带时分秒),这里转一下 lqw.ge(JobDataTrend::getStorageTime, Date.from(start.atStartOfDay(ZoneId.systemDefault()).toInstant())); @@ -133,7 +135,7 @@ public class JobDataTrendServiceImpl extends ServiceImpl del_flag = '0' - and job_fair_title like concat('%', #{query.jobFairTitle}, '%') + and job_fair_title like concat('%', cast(#{query.jobFairTitle, jdbcType=VARCHAR} as varchar), '%') and job_fair_type = #{query.jobFairType} diff --git a/ruoyi-bussiness/src/test/java/com/ruoyi/cms/externalimport/ExternalJobExcelParserTest.java b/ruoyi-bussiness/src/test/java/com/ruoyi/cms/externalimport/ExternalJobExcelParserTest.java new file mode 100644 index 0000000..d0926d2 --- /dev/null +++ b/ruoyi-bussiness/src/test/java/com/ruoyi/cms/externalimport/ExternalJobExcelParserTest.java @@ -0,0 +1,56 @@ +package com.ruoyi.cms.externalimport; + +import org.apache.poi.ss.usermodel.Row; +import org.apache.poi.ss.usermodel.CellStyle; +import org.apache.poi.xssf.usermodel.XSSFWorkbook; +import org.junit.jupiter.api.Test; + +import java.io.ByteArrayInputStream; +import java.io.ByteArrayOutputStream; +import java.time.LocalDate; +import java.util.Date; +import java.util.List; + +import static org.junit.jupiter.api.Assertions.assertEquals; + +class ExternalJobExcelParserTest { + @Test + void parsesExcelDatesAndLocationFallback() throws Exception { + byte[] data = workbookBytes(); + List rows = new ExternalJobExcelParser().parse(new ByteArrayInputStream(data)); + + assertEquals(1, rows.size()); + ExternalJobCandidate row = rows.get(0); + assertEquals("2026-07-14", row.getPostingDate()); + assertEquals("2026-07-20", row.getCollectDate()); + assertEquals("石河子市", row.getLocation()); + assertEquals("Java 工程师", row.getJobTitle()); + } + + private byte[] workbookBytes() throws Exception { + String[] headers = {"Aca112", "Acb22a", "SalaryLow", "SalaryHight", "Aae397", "AAB004", "ORG", "ACE760", "Collect_time", "City"}; + try (XSSFWorkbook workbook = new XSSFWorkbook(); ByteArrayOutputStream output = new ByteArrayOutputStream()) { + Row header = workbook.createSheet("合并数据").createRow(0); + for (int i = 0; i < headers.length; i++) { + header.createCell(i).setCellValue(headers[i]); + } + Row data = workbook.getSheetAt(0).createRow(1); + CellStyle dateStyle = workbook.createCellStyle(); + dateStyle.setDataFormat(workbook.getCreationHelper().createDataFormat().getFormat("yyyy-mm-dd")); + data.createCell(0).setCellValue("Java 工程师"); + data.createCell(1).setCellValue("负责平台开发"); + data.createCell(2).setCellValue(5000); + data.createCell(3).setCellValue(10000); + data.createCell(4).setCellValue(Date.from(LocalDate.of(2026, 7, 14).atStartOfDay(java.time.ZoneId.systemDefault()).toInstant())); + data.getCell(4).setCellStyle(dateStyle); + data.createCell(5).setCellValue("示例企业"); + data.createCell(6).setCellValue("智联招聘"); + data.createCell(7).setCellValue("https://www.zhaopin.com/job/42"); + data.createCell(8).setCellValue(Date.from(LocalDate.of(2026, 7, 20).atStartOfDay(java.time.ZoneId.systemDefault()).toInstant())); + data.getCell(8).setCellStyle(dateStyle); + data.createCell(9).setCellValue("石河子市"); + workbook.write(output); + return output.toByteArray(); + } + } +} diff --git a/ruoyi-bussiness/src/test/java/com/ruoyi/cms/externalimport/ExternalJobFingerprintTest.java b/ruoyi-bussiness/src/test/java/com/ruoyi/cms/externalimport/ExternalJobFingerprintTest.java new file mode 100644 index 0000000..441a22d --- /dev/null +++ b/ruoyi-bussiness/src/test/java/com/ruoyi/cms/externalimport/ExternalJobFingerprintTest.java @@ -0,0 +1,42 @@ +package com.ruoyi.cms.externalimport; + +import org.junit.jupiter.api.Test; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertNotEquals; + +class ExternalJobFingerprintTest { + @Test + void identityFingerprintIgnoresTrackingParametersAndMutableContent() { + ExternalJobCandidate first = candidate("https://www.zhaopin.com/job/42/?utm_source=test&from=feed", "5000", "岗位描述一"); + ExternalJobCandidate changed = candidate("https://www.zhaopin.com/job/42?utm_campaign=summer", "7000", "岗位描述二"); + + assertEquals(ExternalJobFingerprint.identity(first), ExternalJobFingerprint.identity(changed)); + assertNotEquals(ExternalJobFingerprint.content(first), ExternalJobFingerprint.content(changed)); + } + + @Test + void identityFingerprintSeparatesDifferentSources() { + ExternalJobCandidate zhaopin = candidate("https://www.zhaopin.com/job/42", "5000", "岗位描述"); + ExternalJobCandidate liepin = candidate("https://www.zhaopin.com/job/42", "5000", "岗位描述"); + liepin.setSourceCode("liepin"); + + assertNotEquals(ExternalJobFingerprint.identity(zhaopin), ExternalJobFingerprint.identity(liepin)); + } + + private ExternalJobCandidate candidate(String url, String salary, String description) { + ExternalJobCandidate candidate = new ExternalJobCandidate(); + candidate.setSourceCode("zhaopin"); + candidate.setCompanyName("石河子示例企业"); + candidate.setJobTitle("Java 工程师"); + candidate.setLocation("石河子市"); + candidate.setJobUrl(url); + candidate.setSalaryLow(salary); + candidate.setSalaryHigh("10000"); + candidate.setDescription(description); + candidate.setEducationCode("3"); + candidate.setExperienceCode("4"); + candidate.setPostingDate("2026-07-20"); + return candidate; + } +} diff --git a/scripts/apply-external-job-import-test.sh b/scripts/apply-external-job-import-test.sh new file mode 100755 index 0000000..d69acfb --- /dev/null +++ b/scripts/apply-external-job-import-test.sh @@ -0,0 +1,37 @@ +#!/usr/bin/env bash +# 在 SHZ 测试 HighGo 数据库执行外部岗位导入迁移。 +# 仅支持测试环境;迁移 SQL 自身为事务性、可重复执行的 migration。 +set -Eeuo pipefail + +SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd -P)" +PROJECT_DIR="$(cd -- "$SCRIPT_DIR/.." && pwd -P)" +MIGRATION_FILE="${MIGRATION_FILE:-$PROJECT_DIR/sql/migration_external_job_import.sql}" +PASSWORD_FILE="${HIGHGO_TEST_PASSWORD_FILE:-${HIGHGO_PASSWORD_FILE:-$PROJECT_DIR/.codex/skills/query-shz-highgo/.password}}" +TEST_SSH_HOST="${HIGHGO_TEST_SSH_HOST:-root@47.111.103.66}" +TEST_PSQL_PATH="${HIGHGO_TEST_PSQL_PATH:-/opt/highgo/hgdb-4.5/bin/psql}" +TEST_DB_HOST="${HIGHGO_TEST_DB_HOST:-127.0.0.1}" +TEST_DB_PORT="${HIGHGO_TEST_DB_PORT:-5866}" +TEST_DB_NAME="${HIGHGO_TEST_DB_NAME:-highgo}" +TEST_DB_USER="${HIGHGO_TEST_DB_USER:-sysdba}" + +[[ "${CONFIRM_TEST_MIGRATION:-}" == "yes" ]] || { + echo '为防误操作,请使用 CONFIRM_TEST_MIGRATION=yes 执行测试数据库迁移。' >&2 + exit 2 +} + +[[ -r "$MIGRATION_FILE" ]] || { echo "迁移文件不可读: $MIGRATION_FILE" >&2; exit 2; } +[[ -r "$PASSWORD_FILE" ]] || { echo '测试数据库密码文件不可读。' >&2; exit 2; } + +for command in ssh; do + command -v "$command" >/dev/null 2>&1 || { echo "缺少必要命令: $command" >&2; exit 2; } +done + +printf '执行测试库迁移:%s\n' "$(basename "$MIGRATION_FILE")" +{ + cat "$PASSWORD_FILE" + printf '\n' + cat "$MIGRATION_FILE" +} | ssh -o BatchMode=yes -o ConnectTimeout=20 "$TEST_SSH_HOST" \ + "IFS= read -r PGPASSWORD; export PGPASSWORD; exec '$TEST_PSQL_PATH' -X --no-psqlrc -h '$TEST_DB_HOST' -p '$TEST_DB_PORT' -U '$TEST_DB_USER' -d '$TEST_DB_NAME' -v ON_ERROR_STOP=1 -f -" + +printf '测试库迁移已完成。请使用只读查询脚本验证结果。\n' diff --git a/sql/migration_external_job_import.sql b/sql/migration_external_job_import.sql new file mode 100644 index 0000000..a8589a8 --- /dev/null +++ b/sql/migration_external_job_import.sql @@ -0,0 +1,221 @@ +-- 外部 Excel 岗位全量快照同步。 +-- 执行目标:HighGo/PostgreSQL 的 shz schema。 +-- 本脚本为增量 migration;不删除现有 job、company、row_work 或监测数据。 + +BEGIN; + +-- 串行化本 migration 的并发执行,避免两个发布进程同时计算 MAX(id)+1。 +-- 锁仅持续到本事务提交;重复执行不会改变已存在的业务记录或 Quartz 状态。 +SELECT pg_advisory_xact_lock(hashtext('shz.migration_external_job_import')); + +ALTER TABLE "shz"."job" + ADD COLUMN IF NOT EXISTS "external_import_flag" CHAR(1) NOT NULL DEFAULT '0', + ADD COLUMN IF NOT EXISTS "external_source_code" VARCHAR(64), + ADD COLUMN IF NOT EXISTS "external_source_name" VARCHAR(200), + ADD COLUMN IF NOT EXISTS "job_fingerprint" VARCHAR(64), + ADD COLUMN IF NOT EXISTS "job_content_fingerprint" VARCHAR(64), + ADD COLUMN IF NOT EXISTS "external_batch_id" VARCHAR(64), + ADD COLUMN IF NOT EXISTS "external_first_seen_at" TIMESTAMP, + ADD COLUMN IF NOT EXISTS "external_last_seen_at" TIMESTAMP; + +COMMENT ON COLUMN "shz"."job"."external_import_flag" IS '是否外部 Excel 导入岗位:0否,1是'; +COMMENT ON COLUMN "shz"."job"."job_fingerprint" IS '外部岗位身份指纹:FP_V1 + 来源编码 + 标准化岗位链接'; +COMMENT ON COLUMN "shz"."job"."job_content_fingerprint" IS '外部岗位内容指纹:用于同一身份岗位的原地更新'; + +CREATE TABLE IF NOT EXISTS "shz"."job_import_batch" ( + "batch_id" VARCHAR(64) PRIMARY KEY, + "file_id" VARCHAR(64) NOT NULL, + "file_name" VARCHAR(500) NOT NULL, + "file_sha256" VARCHAR(64) NOT NULL, + "snapshot_scope" TEXT, + "fingerprint_version" VARCHAR(16) NOT NULL, + "status" VARCHAR(20) NOT NULL, + "total_rows" INTEGER NOT NULL DEFAULT 0, + "valid_rows" INTEGER NOT NULL DEFAULT 0, + "invalid_rows" INTEGER NOT NULL DEFAULT 0, + "duplicate_rows" INTEGER NOT NULL DEFAULT 0, + "insert_count" INTEGER NOT NULL DEFAULT 0, + "update_count" INTEGER NOT NULL DEFAULT 0, + "keep_count" INTEGER NOT NULL DEFAULT 0, + "restore_count" INTEGER NOT NULL DEFAULT 0, + "delete_count" INTEGER NOT NULL DEFAULT 0, + "error_summary" TEXT, + "created_at" TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP, + "updated_at" TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP, + "completed_at" TIMESTAMP +); + +CREATE TABLE IF NOT EXISTS "shz"."job_import_row" ( + "row_id" BIGINT GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY, + "batch_id" VARCHAR(64) NOT NULL, + "excel_row_number" INTEGER NOT NULL, + "source_name" VARCHAR(200), + "source_code" VARCHAR(64), + "job_fingerprint" VARCHAR(64), + "content_fingerprint" VARCHAR(64), + "validation_status" VARCHAR(20) NOT NULL, + "result_status" VARCHAR(20) NOT NULL, + "error_message" TEXT, + "raw_payload" TEXT, + "job_id" BIGINT, + "created_at" TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP +); + +CREATE TABLE IF NOT EXISTS "shz"."job_import_source_alias" ( + "alias_name" VARCHAR(200) PRIMARY KEY, + "source_code" VARCHAR(64) NOT NULL, + "website_id" BIGINT, + "is_active" CHAR(1) NOT NULL DEFAULT '1', + "snapshot_required" CHAR(1) NOT NULL DEFAULT '1', + "remark" VARCHAR(500), + "create_time" TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP, + "update_time" TIMESTAMP +); + +CREATE UNIQUE INDEX IF NOT EXISTS "uk_job_external_active_fingerprint" + ON "shz"."job" ("job_fingerprint") + WHERE "external_import_flag" = '1' AND "del_flag" = '0'; + +CREATE INDEX IF NOT EXISTS "idx_job_import_batch_sha_status" + ON "shz"."job_import_batch" ("file_sha256", "status"); +CREATE INDEX IF NOT EXISTS "idx_job_import_row_batch" + ON "shz"."job_import_row" ("batch_id", "excel_row_number"); +CREATE INDEX IF NOT EXISTS "idx_job_external_source_active" + ON "shz"."job" ("external_source_code", "external_import_flag", "del_flag"); + +-- Quartz 任务默认暂停,防止尚未配置 external-job-import URL/token 时发生无效拉取。 +-- 完成环境配置后,在“定时任务”中启用即可;计划为每周三、周六 03:00。 +INSERT INTO "shz"."sys_job" ( + "job_id", "job_name", "job_group", "invoke_target", "cron_expression", + "misfire_policy", "concurrent", "status", "create_by", "create_time", + "update_by", "update_time", "remark" +) +SELECT + (SELECT COALESCE(MAX("job_id"), 0) + 1 FROM "shz"."sys_job"), + '外部岗位全量同步', + 'DEFAULT', + 'jobCron.importLatestExternalSnapshot()', + '0 0 3 ? * WED,SAT', + '3', + '1', + '1', + 'system', + CURRENT_TIMESTAMP, + '', + NULL, + '外部 Excel 全量快照同步;配置 external-job-import 后由管理员启用' +WHERE NOT EXISTS ( + SELECT 1 FROM "shz"."sys_job" WHERE "invoke_target" = 'jobCron.importLatestExternalSnapshot()' +); + +-- 为样例及当前采集范围初始化来源管理记录。名称相近的来源不自动合并。 +INSERT INTO "shz"."website_management" ("website_name", "website_url", "is_active", "data_type", "del_flag", "create_by", "create_time", "remark") +SELECT seed.website_name, '', '1', '2', '0', 'system', CURRENT_TIMESTAMP, '外部 Excel 岗位采集来源' +FROM (VALUES + ('石河子招聘网'), ('智联招聘'), ('天互招聘网'), ('猎聘网'), ('乌鲁木齐快聘网'), ('前程无忧') +) AS seed(website_name) +WHERE NOT EXISTS ( + SELECT 1 FROM "shz"."website_management" existing + WHERE existing."website_name" = seed.website_name AND existing."del_flag" = '0' +); + +-- 以 alias_name 为幂等键;若此前仅完成了部分迁移,重跑时会补齐网站关联。 +INSERT INTO "shz"."job_import_source_alias" AS target ("alias_name", "source_code", "website_id", "is_active", "snapshot_required", "remark") +SELECT seed.alias_name, seed.source_code, website."website_id", '1', '1', '外部 Excel ORG 来源映射' +FROM (VALUES + ('石河子招聘网', 'shz_job_net'), + ('智联招聘', 'zhaopin'), + ('天互招聘网', 'tianhu_job'), + ('猎聘网', 'liepin'), + ('乌鲁木齐快聘网', 'wlmq_fast_job'), + ('前程无忧', '51job') +) AS seed(alias_name, source_code) +LEFT JOIN "shz"."website_management" website ON website."website_name" = seed.alias_name AND website."del_flag" = '0' +ON CONFLICT ("alias_name") DO UPDATE +SET "source_code" = EXCLUDED."source_code", + "website_id" = EXCLUDED."website_id", + "is_active" = EXCLUDED."is_active", + "snapshot_required" = EXCLUDED."snapshot_required", + "remark" = EXCLUDED."remark", + "update_time" = CURRENT_TIMESTAMP +WHERE target."source_code" IS DISTINCT FROM EXCLUDED."source_code" + OR target."website_id" IS DISTINCT FROM EXCLUDED."website_id" + OR target."is_active" IS DISTINCT FROM EXCLUDED."is_active" + OR target."snapshot_required" IS DISTINCT FROM EXCLUDED."snapshot_required" + OR target."remark" IS DISTINCT FROM EXCLUDED."remark"; + +-- 四字菜单:来源管理、入库监测、采集趋势、统计分析。 +INSERT INTO "shz"."sys_menu" ( + "menu_id", "menu_name", "parent_id", "order_num", "path", "component", "query", "route_name", + "is_frame", "is_cache", "menu_type", "visible", "status", "perms", "icon", "create_by", "create_time", "remark" +) +SELECT (SELECT COALESCE(MAX("menu_id"), 0) + 1 FROM "shz"."sys_menu"), + '招聘信息采集监测分析', 0, 99, 'recruit-monitor', '', '', 'RecruitMonitor', + 1, 0, 'M', '0', '0', 'cms:recruit:root', 'chart', 'system', CURRENT_TIMESTAMP, '外部岗位采集监测分析根菜单' +WHERE NOT EXISTS (SELECT 1 FROM "shz"."sys_menu" WHERE "perms" = 'cms:recruit:root'); + +INSERT INTO "shz"."sys_menu" ( + "menu_id", "menu_name", "parent_id", "order_num", "path", "component", "query", "route_name", + "is_frame", "is_cache", "menu_type", "visible", "status", "perms", "icon", "create_by", "create_time", "remark" +) +SELECT ids."base_id" + ROW_NUMBER() OVER (ORDER BY seed.order_num), + seed.menu_name, parent."menu_id", seed.order_num, seed.path, seed.component, '', seed.route_name, + 1, 0, 'C', '0', '0', seed.perms, 'table', 'system', CURRENT_TIMESTAMP, seed.remark +FROM (VALUES + ('来源管理', 1, 'recruit/source', 'Recruit/Source/index', 'RecruitSource', 'cms:recruit:source:list', '外部岗位来源与别名管理'), + ('入库监测', 2, 'recruit/monitor', 'Recruit/Monitor/index', 'RecruitMonitor', 'cms:recruit:monitor:list', '外部岗位导入批次监测'), + ('采集趋势', 3, 'recruit/trend', 'Recruit/Trend/index', 'RecruitTrend', 'cms:recruit:trend:list', '外部岗位采集趋势'), + ('统计分析', 4, 'recruit/analysis', 'Recruit/Analysis/index', 'RecruitAnalysis', 'cms:recruit:analysis:list', '外部岗位统计分析') +) AS seed(menu_name, order_num, path, component, route_name, perms, remark) +JOIN "shz"."sys_menu" parent ON parent."perms" = 'cms:recruit:root' +CROSS JOIN (SELECT COALESCE(MAX("menu_id"), 0) AS "base_id" FROM "shz"."sys_menu") ids +WHERE NOT EXISTS (SELECT 1 FROM "shz"."sys_menu" existing WHERE existing."perms" = seed.perms); + +INSERT INTO "shz"."sys_menu" ( + "menu_id", "menu_name", "parent_id", "order_num", "path", "component", "query", "route_name", + "is_frame", "is_cache", "menu_type", "visible", "status", "perms", "icon", "create_by", "create_time", "remark" +) +SELECT (SELECT COALESCE(MAX("menu_id"), 0) + 1 FROM "shz"."sys_menu"), + '手动导入', parent."menu_id", 1, '', '', '', '', 1, 0, 'F', '0', '0', + 'cms:recruit:import:manual', '#', 'system', CURRENT_TIMESTAMP, '手动拉取并导入最新外部 Excel' +FROM "shz"."sys_menu" parent +WHERE parent."perms" = 'cms:recruit:monitor:list' + AND NOT EXISTS (SELECT 1 FROM "shz"."sys_menu" WHERE "perms" = 'cms:recruit:import:manual'); + +INSERT INTO "shz"."sys_menu" ( + "menu_id", "menu_name", "parent_id", "order_num", "path", "component", "query", "route_name", + "is_frame", "is_cache", "menu_type", "visible", "status", "perms", "icon", "create_by", "create_time", "remark" +) +SELECT (SELECT COALESCE(MAX("menu_id"), 0) + 1 FROM "shz"."sys_menu"), + '来源维护', parent."menu_id", 1, '', '', '', '', 1, 0, 'F', '0', '0', + 'cms:recruit:source:edit', '#', 'system', CURRENT_TIMESTAMP, '维护外部来源和来源别名' +FROM "shz"."sys_menu" parent +WHERE parent."perms" = 'cms:recruit:source:list' + AND NOT EXISTS (SELECT 1 FROM "shz"."sys_menu" WHERE "perms" = 'cms:recruit:source:edit'); + +INSERT INTO "shz"."sys_menu" ( + "menu_id", "menu_name", "parent_id", "order_num", "path", "component", "query", "route_name", + "is_frame", "is_cache", "menu_type", "visible", "status", "perms", "icon", "create_by", "create_time", "remark" +) +SELECT (SELECT COALESCE(MAX("menu_id"), 0) + 1 FROM "shz"."sys_menu"), + '监测维护', parent."menu_id", 2, '', '', '', '', 1, 0, 'F', '0', '0', + 'cms:recruit:monitor:edit', '#', 'system', CURRENT_TIMESTAMP, '维护导入监测记录' +FROM "shz"."sys_menu" parent +WHERE parent."perms" = 'cms:recruit:monitor:list' + AND NOT EXISTS (SELECT 1 FROM "shz"."sys_menu" WHERE "perms" = 'cms:recruit:monitor:edit'); + +INSERT INTO "shz"."sys_role_menu" ("role_id", "menu_id") +SELECT role."role_id", menu."menu_id" +FROM "shz"."sys_role" role +JOIN "shz"."sys_menu" menu ON menu."perms" IN ( + 'cms:recruit:root', 'cms:recruit:source:list', 'cms:recruit:monitor:list', + 'cms:recruit:trend:list', 'cms:recruit:analysis:list', 'cms:recruit:import:manual', + 'cms:recruit:source:edit', 'cms:recruit:monitor:edit' +) +WHERE role."role_key" = 'admin' + AND NOT EXISTS ( + SELECT 1 FROM "shz"."sys_role_menu" existing + WHERE existing."role_id" = role."role_id" AND existing."menu_id" = menu."menu_id" + ); + +COMMIT;