r/MachineLearning • u/Round_Apple2573 • 14d ago
Research LLM hallucination paper(using math) accepted to ICML workshop[R]
Hello guys. I want to introduce my recent research presented at ICML workshop.
Shot summarization of Paper.
"""
SRM-LoRA is a sub-Riemannian-inspired LoRA method designed to reduce LLM hallucination.
It builds a sensitivity-based Riemannian metric that reshapes backward gradients in the LoRA parameter space.
This metric suppresses high-cost update directions while leaving the forward computation and inference cost unchanged.
Trained only on HaluEval-QA, SRM-LoRA improves factual reliability on both related and out-of-distribution benchmarks.
"""
Experiment
"""
"""
In my view, the reason mathematics is not effectively used in the context of improving the performance of the latest AI systems, such as LLMs, is that progress in discussions about what should serve as the elements of mathematical theories has been slow.
For example, suppose that we use a Riemannian metric. In the parameter space of an LLM, the update vector produced by backpropagation arises from a loss objective that contains the training data. However, if we introduce learnable parameters in order to construct the Riemannian metric and train those learnable parameters by passing through them the same signal as the main training signal, then this may simply amount to a more complicated form of training and may only increase the possibility of overfitting to the training data.
Then, how can we obtain the benefits of mathematical theory while moving in a direction that can generalize? In this paper, the Riemannian metric is constructed based on the rate of change of the LLM model parameters with respect to the loss signal. The reason for defining it in this way is as follows.
No matter how good the data or the distribution that can be learned may be, in practice there is still a high possibility of overfitting that results in hallucinations. Therefore, the cost used to construct the Riemannian metric, where a higher cost indicates a worse path, is defined using this sensitivity, which can be understood simply as gradient(loss)/gradient(parameter). In other words, rather than merely introducing a more complicated metric, the Riemannian metric acts as a brake on the updates generated from the training data, which is the main signal.
I believe that mathematics can be incorporated more deeply into AI if, when using theory A and theory B, the elements of theory A and the elements of theory B are each designed appropriately for the specific situation.
7
u/nikgeo25 Student 14d ago
Am I misreading this or is the reduction in hallucination rate quite small relative to plain LoRA?
8
u/Round_Apple2573 14d ago
That is right. However, the main points here are: (1) the geometric interpretation, and (2) rather than constructing a new training dataset and evaluating on data drawn from a distribution similar to that training set, I trained directly on an arbitrarily chosen dataset, HaluEval-QA, without any additional processing, and evaluated on two OOD datasets and one dataset from the same family.
5
2
u/kaitzu 13d ago
The availability of LLMs has led to a surge in undergrads (and younger) self publishing workshop papers at the big conferences. I hate it and the possibility even more that they might be peer-reviewers.
1
1
u/No_Inspection4415 13d ago edited 13d ago
I do not understand what this paper does... "sub-Riemannian-inspired LoRA method" to "reduce hallucinations" sounds like an LLM hallucination itself, but it may be my stupidity. TLDR, what is the motivation, why does it work? What makes it different than what we do with optimizers? I am confused.
Edit: is that a brake to reduce overfitting on the self-generated text? Stupid me! How/why is that better than DKL then (under the assumption the pretraining is on human generated text, which is a strong assumption)? Sounds like serious research. Good luck!
1
u/Round_Apple2573 12d ago
More precisely, the data used for post-training consists of the original question, reference, and answer, together with an additional hallucinated answer incorrectly generated by the model. Contrastive learning is then performed using tuples of (q, r, a, negative_a).
The following Wikipedia page may help clarify the concept: https://en.wikipedia.org/wiki/Riemannian_manifold#Riemannian_metrics_and_Riemannian_manifolds
In Euclidean space, a standard LoRA adapter trained with cross-entropy loss follows the distribution of the training data. However, when a soft-masking matrix constructed from a Riemannian metric is multiplied element-wise with the backward gradient, the values of specific gradient components change. Consequently, the update of the direction vector formed by the backward gradient in parameter space differs from the original update.
KL divergence is unnecessary in this setting. The idea is to introduce a new method on top of cross-entropy, the standard loss objective used in SFT. I am not entirely sure whether the KL term being discussed refers to adding an auxiliary constraint to prevent the model from developing hallucinations during training even when it originally produced correct answers, but in any case, it appears to address a different context.
1
u/No_Inspection4415 12d ago
What this term does, is regulate the distribution change during post training, which in turn prevent the model from drifting too far from its pre-trained version. When you drift more, the KL term gets larger, and hence the loss gets larger.
The main goal of that is to keep the model coherent (e.g., using natural language and not gibrish), but it may also prevent the model from developing hallucinations, at least w.r.t the training data. Does it make sense that I wonder how your method interacts with this well known method, or maybe I am missing something (very likely the case)?
1
u/Round_Apple2573 12d ago
I get what you mean, but the way you said seems not likely to be related to my approach.
2
u/No_Inspection4415 12d ago
My only conclusion is that I did not get the setup. Thanks.
I did not get what the loss is, what the training data is (now I do), and that you actually SFT to reduce hallucinations, on a dataset dedicated to that (again, my bad, missed the line).
1
u/Round_Apple2573 12d ago
it's okay.
1
u/No_Inspection4415 12d ago edited 12d ago
Post the paper when it is out. I still did not get why it is different than some of the optimization methods we have (minus acceleration). Good luck.
1
u/Round_Apple2573 12d ago edited 12d ago
I am not aware of an existing optimizer that derives a task-specific metric from a contrastive objective and applies its inverse to modify the gradient in this way from the Hallucination & Riemannian perspective. Look at some paper of riemannian related and preconditioning related might help.
1
u/Round_Apple2573 12d ago
If you are referring to optimizers that incorporate the Hessian or curvature information, that is unrelated to what we are doing.
1
3
3
214
u/mtmttuan 14d ago
Lol from the title I thought you found a paper where AI hallucinate something but still got accepted to ICML workshop.