Skip to content
YIZE
P.007 / PROJECT2024

面向多需求的生物数据处理软件Bio-Data Processing · From HMP to Analysis-Ready Matrices

Bio-Data Processing · From HMP to Analysis-Ready Matrices面向多需求的生物数据处理软件

GWAS 里不起眼的那一半工作的登记软件(软著 2024SR1711525):hmp 转纯数阵、数字矩阵插值、正定分块协方差矩阵、定比随机分组。

A registered utility (软件著作权 2024SR1711525) for the unglamorous half of GWAS: HMP-to-matrix conversion, imputation, positive-definite block covariance matrices, and proportional random grouping.

角色Role
开发成员Developer
类型Type
软件Software
年份Year
2024
  • Python
  • Bioinformatics
  • GWAS preprocessing
Software Screenshot
P.007-A
Software Screenshot · placeholder
目录Contents

被跳过的那一半工作

从一份原始基因型文件,到模型真正能读进去的矩阵之间,堆着一层没人写进论文、但每个人都要重做一遍的活。全基因组关联分析(GWAS)这几年是研究热点,可针对「大量 SNP 数据」的处理与插值并没有现成的应用:用得最多的 PLINK 1 有自己的格式约定,和下游模型的数据模型之间隔着一段;另有一些模型则要求先把高相关矩阵分块、再从里面取出正定的那些,而 PLINK 并没有直接做这件事的方法。

面向多需求的生物数据处理软件(登记号 2024SR1711525,软著登字第 14115398 号,2024 年 11 月 6 日登记,著作权人中国农业大学)把这一层做成了四个按钮,而不是一段换个人就要重写一遍的脚本。

四个操作

  1. hmp 转纯数阵。 读入 .hmp 文件,按「空缺 / 隐性 / 显性」的判定转成 0 / 1 / 2,导出无表头的纯数字矩阵——下游模型要的就是这个形状。
  2. 数字矩阵插值。 分行、列、行列三种模式:统计对应方向(或两个方向之积)上 1 与 2 的比例,按这个比例把矩阵里的空位 0 随机补成 1 或 2。方法本身是朴素且好用的随机数插值,好在能用、也说得清楚。
  3. 正定分块协方差矩阵获取。 按协方差(r²)把高相关的矩阵分块,并尽量多地取出其中正定的那些。
  4. 定比随机分组。 除表头外按给定比例把行随机分成两个文件,两个文件都继承原表头——测试集与验证集就是这么切出来的。

按使用手册里的定位,这四个操作分别对接下游的几类用法:0/1/2 生成的相关系数矩阵(Ld 矩阵)用作 Ref、Map 一类生物数学模型的输入;插值后的矩阵用于随机森林等分类模型估计表型;正定分块综合了前两者,服务于那些要求 Ld 矩阵正定的模型;定比随机分组用来切出测试集与验证集,测模型性能。

「正定分块」为什么占了过半的篇幅

因为它正是 PLINK 做不了、而绕开做又会出事的那一件:

  • PLINK 本身没有直接完成这个需求的方法,而且速度较慢;
  • 直接用它取分块矩阵,会不可避免地把不同分块之间的协方差矩阵也一并取出来,最后那个文件大到难以从里面挑出真正需要的东西;
  • 我们把「提取矩阵」这一步交给 python 处理,而不是多开几个 PLINK 进程——这一点差别直接落在内存与 CPU 占用上。

流程是一个循环,不是一条流水线:读入原始碱基信息 → 提取 → 用 PLINK / PLINK2 得到分块 → 用 python 算出分块矩阵 → 用 numpy 筛正定比例 → 达标就停;未达标则把非正定的那些挑出来,据此生成「再次分块」的碱基信息,回到开头,直到总的正定比例达到阈值。结果落在当前目录 result/chunks/ 下以 .vars 结尾的文件里。

既然是循环、又带阈值,那个阈值就得由用户自己用滑动条给定——这个参数没有「正确值」,它取决于下游模型能接受多少不满足条件的矩阵。

为什么它是一个软件,而不是几个脚本

这四个操作各写一遍是一个下午,各自维护一年是一次后悔:三个月后有人重跑,拿到的版本和论文里那一版对不上。把它们收拢成一个行为固定的登记软件,差别就在这里——每次跑出来的是同样的东西,而且下一个人不必再去翻「当初生成这个矩阵的那段脚本」。

使用手册里另外写进了两条工程上的自律,值得留在页面上:只支持标准格式的 hmp 与 csv(以半角逗号分隔),格式不对就报错,不猜;处理过程中 CPU 会被大量占用、界面可能卡顿,这是正常现象而不是死掉。

环境

Python 环境即可。界面用 PySide6,另外用到 numpy、pandas、tqdm、chardet 等常规包。只有第三个功能需要 PLINK 与 PLINK2——不用它就不必下载。Windows、Linux、macOS 都可以跑。

因为程序对原始数据只做读取,中途关掉窗口不会动到原文件。

这一页不写的

用它跑过的那批数据。数据属于产出它们的研究,不属于这个工具。

目录Contents

Overview

Between a raw genotype file and a model sits a pile of conversions that nobody publishes and everybody redoes: PLINK's format conventions do not match the data models of the models downstream, SNP matrices need imputation before analysis, some methods require the correlation structure to be block-partitioned into positive-definite pieces, and every cross-validation needs a grouped split that respects proportions.

面向多需求的生物数据处理软件 (registered 2024SR1711525, 6 November 2024, 著作权人 中国农业大学) is that pile, turned into four operations that run without an afternoon of scripting.

The four operations

  1. HMP to numeric matrix — the HapMap format becomes a plain numeric matrix, which is the form every downstream model actually wants.
  2. Matrix imputation — missing genotypes filled, so the analysis does not have to decide what to do with holes at model time.
  3. Positive-definite block covariance — highly correlated blocks are partitioned and the positive-definite ones extracted, for the methods that require that structure as an input rather than a hope.
  4. Proportional random grouping — reproducible stratified splits, so the "random" in cross-validation is recorded rather than unrepeatable.

Why the block covariance is half of this page

Because it is the one operation PLINK cannot do and cannot be routed around safely: PLINK has no direct method for it and is slow; asking PLINK for the block matrices directly inevitably drags in the covariances between blocks as well, to the point where the resulting file is too large to pick the useful parts out of; and the extraction is done in Python rather than by opening several PLINK processes, which shows up directly in memory and CPU.

So the flow is a loop rather than a pipeline: read the raw base information, extract, get block information from PLINK and PLINK2, compute the block matrices in Python, sieve for the positive-definite share — and stop if it clears the threshold. If it does not, the non-positive-definite blocks become new base information for a second partitioning pass, and the loop repeats until the total share clears. Results land in result/chunks/ as .vars files. Because it is a loop with a threshold, the threshold is set by a slider: it has no "correct" value, since it depends on how many non-qualifying matrices the downstream model will tolerate.

Why this is a project and not a script

Each of these operations is one afternoon of code and one year of regret: the version someone reran three months later does not match the version in the paper. Packaging them as one registered tool with fixed behaviour is the difference — the operations are the same every time, and the tool's existence means the next person does not go looking for the script that made the matrix.

Environment

A Python environment is all it needs. The interface is PySide6, with numpy, pandas, tqdm and chardet among the usual packages. Only the third operation needs PLINK and PLINK2 — skip that one and there is nothing extra to download. Windows, Linux and macOS all run it.

Because the program only ever reads the input data, closing the window mid-run cannot damage the original files.

What is not on this page

The datasets it was used on. They belong to the studies that produced them, not to the tool.