Methodology
How does SystemAudit decide what counts as a finding?
Every finding is verified against your actual source before it reaches the report. The analysis reads a capped subset of files through GitHub's API, cross-checks each claim against measured facts, and drops anything it cannot tie to a specific file and line. Coverage is reported as a ratio, so a report never implies more was read than actually was.
What is measured
- Secrets and credentials. Pattern detection for API keys, tokens, private keys, database connection strings and committed environment files, each reported with its file, line and a redacted snippet.
- Security patterns. Structural risks in source — string-concatenated queries, unvalidated request input, unescaped rendering, shell invocation reachable from user input, insecure transport.
- Dependencies. Manifests across nine ecosystems, used to establish what the project is built from.
- Engineering posture. Whether tests, CI, linting, type safety, documentation and security tooling are present — and how well evidenced each one is, rather than whether a filename hinted at it.
- Architecture. Component structure and how the parts connect, rendered as a map.
How coverage is reported
The scanner does not clone your repository. It fetches a capped subset of files through GitHub's API — 40, 80 or 150 depending on project size — and every report states how many files were analysed out of how many exist.
That ratio is there so a clean result is legible. “No secrets found in 40 of 900 files” and “no secrets found” are very different statements, and only one of them is honest.
Why findings get dropped
An audit tool that invents problems is worse than no audit tool, because it spends a team's time on work that was never needed. So the analysis is checked against the source rather than trusted: claims that contradict measured facts are corrected, and any issue that cannot be tied to an exact file and line is downgraded or removed before you see it.
Severity and confidence are reported separately, because they answer different questions. Severity is how bad this would be if real; confidence is how likely it is to be real. A tool that collapses them lets a guess carry the weight of a fact.
What this cannot tell you
- It does not find every vulnerability. This is prioritised, surface-level risk discovery — not a penetration test, and not a substitute for a deep human review of a high-stakes system.
- It does not read your entire repository, and the coverage ratio on every report says so explicitly.
- It cannot judge whether your architecture suits where the product is going. That requires someone who understands the domain and the roadmap.
- It does not test whether an attacker could actually break in. Nobody is trying; it is reading, not attacking.
How aggregate research is produced
Scans accumulate into a corpus that can be reported on in aggregate. Two rules govern what is ever published from it, and both are enforced in code rather than by intention.
- Nothing publishes below 100 scans. The gate is the sample size, not a date. A statistic drawn from a handful of repositories and presented as an industry finding is an unearned claim.
- No scan is attributable to a repository. Private repositories are excluded outright. Nothing published carries a repo name, owner, URL, file path or line number — only counts. Any bucket holding fewer than 10 scans is suppressed rather than reported, because a percentage over a handful of projects is re-identification with extra steps.
Frequently asked questions
Does SystemAudit read my whole repository?
No, and no report implies otherwise. The scanner fetches a capped subset of files through GitHub's API — 40, 80 or 150 depending on project size — and never clones. Every report states coverage as a ratio of files analysed to files present, so you can see exactly how much was read.
How do you avoid reporting problems that are not there?
Every finding is checked against the actual source before it reaches you. The analysis cross-references claims against measured facts — real test-file counts, real configuration, real file contents — corrects anything that contradicts them, and drops any issue it cannot tie to a specific file and line.
What is the difference between severity and confidence?
Severity answers how bad a finding would be if it is real. Confidence answers how likely it is to be real. They are independent: a vendor-issued key format is near-certain, while a rule firing on the absence of a signal in a partial sample is a suggestion. Collapsing the two lets a guess carry the weight of a fact.
Can I verify the detection rules myself?
Yes. The scanner engine is MIT-licensed and public, so what counts as a secret, what counts as a risk, and where the analysis stops are all readable rather than asserted. A security tool asking to be trusted should let you check how it decides.
Read the detection rules
The scanner engine is MIT-licensed and public. What counts as a secret, what counts as a risk, and where the analysis stops are all readable.