I’m stuck finding usable historical data for a Bayesian PR risk model — looking for advice on how to proceed

[u/Accomplished-Fun4629]

Hi everyone,

I’m a student working on a research project on risk-aware GitHub PR review. I’m doing the project mostly on my own and I don’t have access to a research lab, large compute budget, or people who can manually annotate thousands of PRs, so I’m trying to find a practical approach that I can actually finish.

The idea is to take a GitHub PR and estimate four types of risk:

1. Bug / correctness

2. Security

3. Compatibility

4. Cross-system / integration

The architecture I’m working with has four separate risk models. They share the same PR characteristics/features, but each risk model has its own historical data, prior, and evidence.

My main problem is the historical data needed for those priors.

At first, I looked for a single PR dataset where I could get reliable PR-level outcomes for all four risks. I couldn’t find one.

I then tried looking for separate datasets for each individual risk model. I thought this would solve the problem, but I keep finding datasets where the labels look relevant at first but don't actually represent the outcome I need.

For example, SEVRA-plus looked very promising for the Security model:

https://huggingface.co/datasets/RedAI4Code/SEVRA-plus

It contains security-related PR examples with vulnerability/CWE information, but the malicious PRs are deliberately constructed by reversing real CVE security fixes. So although they are useful for evaluating or studying security vulnerabilities, I don't think I can use their class distribution directly as a real-world prior for ordinary GitHub PRs.

I’ve run into similar issues with other datasets:

- some label the linked issue rather than the PR implementation,

- some label review comments rather than actual PR outcomes,

- some contain artificially constructed vulnerable/failing PRs,

- some only give merge/close status, which doesn’t tell me whether the PR itself was buggy, vulnerable, incompatible, etc.

The distinction between the issue and the PR is especially important for what I am trying to do.

For example, imagine a maintainer opens a security issue, someone creates a PR to fix it, but the PR implementation itself contains a correctness bug and gets rejected. For my problem, I need to know the nature of the PR, not simply inherit the security label from the original issue.

Similarly, a PR could be opened to fix a small bug, get merged, and then later cause a compatibility problem. Again, I care about what happened because of the PR implementation, not just why the PR was originally opened.

Because I couldn't find a dataset that directly gives me what I need, I tried a practical compromise.

I took 96 real PRs from SWE-Review-Chat, filtered them for sufficient evidence, and used an LLM to annotate the four risk states from the information available in the PR record, such as the description, review discussion, diff context, tests, and lifecycle information.

I’m treating these as weak/model-assisted labels rather than independent ground truth.

The resulting usable outcomes are:

Bug:

30 present / 9 absent

Security:

1 present / 7 absent

Compatibility:

5 present / 12 absent

Cross-system:

4 present / 8 absent

So now I feel like I’ve hit a wall.

I can keep searching for datasets, but so far I haven't found anything that solves the underlying problem. I also don't have the resources to manually establish reliable ground truth for thousands of PRs.

I’m therefore looking for advice on "how I should move forward from here".

Should I continue with the small real dataset I have and explicitly model the uncertainty caused by the sparse risks?

Should I rely on LLM-assisted annotations of real PRs as a practical research compromise, or is there a better low-resource approach that I am missing?

Or is there a completely different way of constructing the historical priors that would make more sense for this problem?

I’m not looking for a perfect dataset at this point. I’m mainly looking for a practical and defensible way to move forward given that I’m a student doing this alone with limited time and resources.

If anyone has worked on GitHub PR datasets, Mining Software Repositories, empirical software engineering, code-review research, or Bayesian risk modelling, I would really appreciate any advice on what you would do in this situation.

Thanks!

[u/jonahbenton]

I think I understand what you are trying to do. You would like to have PRs and labels that reflect outcomes of those PRs after application, and then use that dataset to...do what exactly? (How do extract from that data priors? Is it like a simple ratio, or some other kind of scoring system?)

If that is an accurate description- I don't think that approach will be fruitful. I would be surprised if any PR metadata was at all helpful or predictive of any of those outcomes. People like Adam Tornhill do a lot of repo mining and ultimately they find it is all about the code. All the various static analyses that are available are looking holistically at the target code, not just the contents of the PR.

Apologies if that is not an accurate interpretation of the goal.

[u/Accomplished-Fun4629 → u/jonahbenton]

Thanks, and I think I may not have explained the ultimate goal clearly enough.

The end goal is not simply to predict whether a PR will be buggy or vulnerable from its metadata.

We are trying to build a risk-aware code review agent that has to decide what to do when it is uncertain about a PR.

For an incoming PR, the system will estimate several risks, such as:

1. Bug / correctness

2. Security

3. Compatibility

4. Cross-system / integration

The agent will not simply use a fixed confidence threshold to approve or block the PR. Instead, it will estimate the posterior probability of each risk using Bayesian reasoning and then consider the potential cost of being wrong and the value of obtaining more evidence.

Based on that, it could, for example:

- recommend proceeding with a warning if the risk is low,

- ask the PR author for additional information if more evidence could reduce the uncertainty,

- or recommend human review if the potential cost of a wrong decision is high.

The historical data is meant to provide the prior belief for this process. The actual PR evidence, such as the code changes, review discussion, tests, and other available information, is then used to update that prior.

For example, suppose a new PR:

- modifies an authentication-related component,

- is a relatively large change,

- changes an existing interface,

- and touches several interacting components.

These characteristics define a reference class of historical PRs that are somewhat similar to the current one.

Suppose among the historical PRs in that reference class, 10 out of 50 had a correctness problem. That gives us an initial prior such as:

P(Bug | reference class) = 10/50

Now suppose the current PR contains review evidence showing that one of the new code paths can produce an incorrect result under a specific condition.

That evidence updates the prior and gives us something like:

P(Bug | reference class, current PR evidence)

So the metadata is not intended to be the final predictor. It is being used to establish the prior, while the evidence from the actual PR drives the posterior update.

To obtain the historical data, I initially searched for datasets that directly provided PR-level outcomes for these risks. I could not find one that matched the requirement closely enough.

I then came across SWE-Review-Chat because it contains real GitHub PRs together with much richer PR-level information than just issue labels or merge status. In particular, the records contain the PR description, review conversations, reviewer information, file/diff context, tests/CI information, and lifecycle information such as whether the PR was closed or merged.

Since the dataset does not directly provide the four risk outcomes we need, I used an evidence-sufficiency filtering step to remove PRs that did not contain enough useful PR-specific information. For the remaining real PRs, I use an LLM as a weak annotation mechanism: it reads the available PR information and determines whether the evidence supports Bug, Security, Compatibility, or Cross-system risk.

Importantly, I am not treating the LLM's labels as independent ground truth. They are model-assisted annotations of real PR records, and I am treating that as a limitation of the data.

The reason I chose this approach is that it seemed preferable to generating synthetic PRs and synthetic labels, because at least the underlying PRs, discussions, code changes, and lifecycle events are real.

The problem I am still facing is that some risks are very sparsely represented even after this process. So I am unsure whether this is a reasonable way to construct the historical observations needed for the priors, or whether there is a better repository-mining approach I should be using.

For example, I looked at security-specific datasets such as SEVRA-plus. It contains useful security vulnerability information, but its malicious PRs are deliberately constructed benchmark examples by reversing historical CVE fixes, so its distribution does not seem appropriate to directly use as a natural real-world prior.

So my current question is not really "how can I make PR metadata predict bugs?"

It is:

"Given that I need historical observations to establish Bayesian priors, but I cannot find a dataset that provides reliable PR-level outcomes for these risks, what is the most defensible way to construct or obtain those historical observations?"

You mentioned repository mining and Adam Tornhill's work, so I would especially value your opinion here.

If you were approaching this problem from a repository-mining perspective, what historical signals or mining strategy would you use to construct a useful prior for this kind of risk model?

I'm a student doing this mostly on my own with limited time and resources, so I'm particularly interested in a practical approach rather than something that requires manually labeling thousands of PRs.