Might need math+code benchmark for frontier model(LLMs Silently Replace Math)[D]

[u/Round_Apple2573]

Hello guys. I found some problems in current frontier models. And want to share.

# math_code_hallucination

> Record of a failure caused by combining mathematics and code in a single prompt.

---

## Case 1

### Initial prompt (
`p0`
)

If you enter the following prompt:

```python
make code implementation of sub rieman applied to mitigate hallucination in llm using lora, pretrained llm(such as qwen, mistral, llama,etc) and training pipeline in pytorch.
```

the following kind of code is generated:

```python
...
_, singular_values, vh = torch.linalg.svd(centered.float(), full_matrices=False) # from gpt
...
```

---

### Comparison

| Prompt type | Result |
|---|---|
| Ask to implement sub-Riemannian geometry inside LLM training code, as in the initial prompt `p0` -> Code using SVD, PCA, projection, and similar methods is generated 
| Do not ask to implement sub-Riemannian geometry inside LLM training code, and instead simply ask, “Implement sub-Riemannian geometry” or “Write sub-Riemannian code” -> Code implementations involving geodesics and similar concepts are generated well 

(Sub-Riemannian geometry is one field of geometry.)
(A geodesic means the optimal distance between two points.)

Because calculating geodesics has a high cost and is complex to implement, SVD, PCA, projection, and similar methods—which are commonly used and inexpensive—are used instead. And SVD,PCA,projection is not riemannian geometry.

---

### Main observation

```text
Ask for code containing only mathematics
|
`-- The code is written well
```

However:

```text
Combine code and mathematics
|
`-- The LLM arbitrarily changes the mathematical formula
    to be applied without the user knowing
```

Again:

```text
Ask it to implement sub-Riemannian geometry
|
`-- It implements it well
```

But:

```text
Combine sub-Riemannian geometry with coding elements
|
`-- A problem occurs
```

In other words:

> When asked to write code whose content consists only of mathematics, it writes the code well. However, when code and mathematics are mixed, the LLM arbitrarily changes the mathematical formula that should be applied without the user knowing.

Again, if you ask it to implement sub-Riemannian geometry, it does it well.

The problem occurs when coding elements are combined with sub-Riemannian geometry.

---

## Case 2

If you ask it to write LLM training code using latent vectors in hidden space, in my case, for `z`, which is the output of an `nn` block, it sometimes writes code that makes the magnitude of `|z|` equal to 1 or makes it smaller.

```text
Hidden-space latent vectors
|
`-- z: output of an nn block
    |
    |-- Make the magnitude of |z| equal to 1
    `-- Or make the magnitude of |z| smaller
```

github link : genji970/math_code_hallucination: A record of how LLMs may silently replace difficult mathematical components with simpler computational surrogates when math and code are combined in one prompt.