Every field faces large data; genetics faces more of it than most. A GWAS
genotype matrix is routinely tens of thousands of rows by hundreds of thousands
of columns, and the usual approach — read it all into memory, then compute —
fails at the first step on an ordinary machine. That is not a matter of the
algorithm being slow. The data does not fit.
Common PCA tooling adds four more barriers: a high floor to entry (write code,
set up an environment), no feedback while it runs, no visualisation (the
reduction finishes, then you plot it separately), and rigid handling of how many
components to keep. Those four turn a tool into a black box for anyone who did
not build it.
This software addresses exactly two things: making large data computable, and
making the computation visible. Registered as 2024SR1708999, rights holder
中国农业大学.
- Checks that the file's format is valid, and reads it in blocks.
- Runs on CPU or GPU (GPU via CuPy).
- Creates multiple tasks, each running PCA over a different set of columns.
- Chooses the number of components three ways: by variance ratio, by
maximum-likelihood estimation, or manually.
- Traces the computation and its result visually, while it runs.
- Writes the reduced data back in the same format.
- Reports errors as they happen.
The file is read with Python, numpy or cupy is imported according to the user's
choice, and the data is read block by block. Safety mode runs two checks first:
is the file a proper CSV (column names on the first row, comma-separated), and
are the chosen columns actually floats that can be reduced? The second check
looks redundant until the first table that mixes an ID column, a date column and
some gaps — which is most of them.
A column-name row shorter than fifty thousand entries is shown in full; beyond
that only the first fifty thousand are displayed, to keep the interface
responsive, and everything after the fifty-thousandth counts as always selected
by the last task. That is not an elegant rule. It is a rule that stops the
window from freezing, which is why it is written down here.
The program uses Python generators to run PCA for several tasks within a
single read of the file. Tasks are added one at a time and each one picks its
own columns from the selector; no two tasks may take the same column, because a
column reduced twice cannot be compared against itself. Search and select-all
sit above the selector, which for a wide matrix is a requirement rather than a
convenience.
The three ways to fix the number of components correspond to three kinds of
person who know different things:
- Variance ratio — enter a variance threshold and a number of blocks to
inspect; the program reads that many blocks and computes the smallest
number of components that reaches the threshold. Zero means read to the end of
the file.
- Maximum likelihood — enter how many blocks to inspect; the number of
components follows from the maximum-likelihood estimate.
- Manual — type a number.
The progress bar has three stages, and not for decoration: each stage fails for
a different reason.
| Stage | Progress | What is happening |
| --- | --- | --- |
| Preprocessing | 0–30% | In safety mode, check that each task's columns are floats; compute each task's component count from its setting |
| PCA | 30–50% | Read block by block onto CPU or GPU and fit; after each block the current components are pushed to the front end |
| Transform | 50–100% | Apply the fitted result to the original data and write the output beside it |
Parameters are handed to ECharts during the run, so the charts update live.
"Missing process feedback" is answered directly rather than being waved away
with a please-wait message.
A front end and a back end. The front end — Vue, Vite and naive-ui — holds the
interface, the interaction and the input; the back end — Python and PyQt — holds
the algorithm and the messaging. One entry file runs both. Every button on the
front end corresponds to an action at the back: "import file" really does open
the back end's file dialog, and once the user starts the run, what the front end
collected is handed over for computation.
What is not on this page
Screenshots. The software is registered with the Copyright Protection Center of
China, and the registration number above is what makes it checkable; a picture of
the interface adds nothing to that.