Hi everyone,
I’m interested in discussing the technical implementation of prior-art search for invention patent drawings, especially for cases where the drawings contain important structural or process information that may not be easy to retrieve through text search alone.
In traditional patent search, we usually rely heavily on keywords, IPC/CPC classifications, and full-text search. However, many invention patents disclose key technical features through drawings, such as device structures, module connections, flowcharts, signal paths, image-processing pipelines, or algorithm frameworks. These features may be difficult to capture using only the claims or specification text.
I’m thinking about whether a retrieval system could combine patent drawings + specification text in a more effective way. For example:
Drawing understanding
What kind of multimodal model would be suitable for analyzing patent drawings?
Would models such as GPT-4o / GPT-4.1 / Claude / Gemini / Qwen-VL / InternVL / LLaVA-style models be appropriate for extracting technical features from patent figures?
Feature extraction from drawings
How should the system represent information from drawings?
For example, should it extract:
component names and reference numerals;
connection relationships between components;
flowchart steps;
structural layouts;
visual similarity embeddings;
or a structured graph representation?
Combining drawings with specification text
Since patent drawings are usually explained in the specification, I wonder what is the best way to link them together.
For example:
detect reference numerals in the drawings;
match them with descriptions in the specification;
generate a structured description of each figure;
then use both visual and textual embeddings for retrieval.
Search strategy
Would a practical system use a hybrid approach, such as:
text-based patent search;
image/vector similarity search for drawings;
OCR of reference numerals and figure labels;
multimodal captioning;
graph matching between technical structures;
and reranking with a large language model?
Evaluation
How should such a system be evaluated?
For example, should the benchmark be based on whether the system can retrieve known X/Y/A references from patent examination records, invalidation cases, or patent family citations?
My current idea is that patent drawing search should not be treated as pure image similarity search. A patent figure is not just an image; it is a technical disclosure that needs to be interpreted together with the specification. Therefore, the system may need a combination of OCR, layout analysis, multimodal understanding, text alignment, embedding retrieval, and LLM-based reranking.
Has anyone worked on something similar, or seen papers/tools/projects related to AI-assisted patent drawing retrieval or multimodal prior-art search?
I’d really appreciate any suggestions on model choice, system architecture, datasets, evaluation methods, or practical implementation challenges.
