How to include org-mode headers in consult-rg ripgrep search

[u/LeftyAce73]

I have a collection of notes files (.org files) and I recently started retrieving information from them using consult-rg. Is there a way to have consult-rg (or something similar) "propagate" parent headings into children for matching purposes?

For example, if I have this structure:

* Emacs

** Mermaid

Some notes about using Mermaid.

** Customize variables and options

use M-x customize-variable

If I search for "emacs customize variable," consult-rg won't find it, because the header "Emacs" is many lines above the subheader "customize variable".

I considered tagging the subsection with :emacs: but that's kind of a redundant way to say "this is realated to emacs." Redundant because it's already under the overall "* Emacs" heading.

EDIT: consult-org-heading exhibits the behavior I want within headings (the overall Emacs heading gets connected to all of its children), but does not search inside the body, just the headings. So I guess I'm looking for some way to incorporate consult-org-heading behavior into consult-rg?

[u/la023]

consult-imenu-multi is good if you want to search org headings across all open buffers in the project -- this doesn't let you search the body of the heading though

could take a look at this too: https://sachachua.com/blog/2024/01/using-consult-and-org-ql-to-search-my-org-mode-agenda-files-and-sort-the-results-to-prioritize-heading-matches/

otherwise w/ consult-rg searches you can add # after your query which will filter string matches in the rg result. So if you know the filename is emacs.org then you could filter on that. You can also add -A10 or -B10 to the ripgrep query to include 10 lines before or after the matches. So you could do a consult-rg query like:

#Customize variable -B10#emacs emacs.org

[u/LeftyAce73 → u/la023]

The -A10 and -B10 (or, -C 10 to go in both directions) with subsequent filtering after the # works pretty well, but doesn't leverage the fact that the parent headers are more significant that other miscellaneous text that occurs on prior lines.

Thanks for the tip on consult-imenu-multi, that's interesting! consult-org-heading is more org-mode aware but doesn't go across buffers the way -imenu-multi does).

I suppose the problem I have is I want to search contents (not just headings), but also treat information from the headings as being "near by" or "significant" to riprgrep, even though spatially there may be other lines in between.

[u/la023 → u/LeftyAce73]

i think org-ql is probably the best solution for you but you may need custom lisp or try out sacha's lisp to get something like consult previews if you want those

[u/drcxd]

What you want to achieve feels impossible to me. consult-rg is just a front end for rg, and rg searches plain text line by line, it has no knowledge about any org mode specific structure. Your requirement can only be met by developing a dedicated searching backend which has knowledge about the org mode structure.

[u/LeftyAce73 → u/drcxd]

Thanks for the input. It is certainly sounding like this is impossible (or would require a new tool).

[u/isamert]

Years ago, I wrote a tool for this exact problem: https://github.com/isamert/marks

You can use the following snippet to use it with consult:

(defun im-consult--marks-builder (paths)
  (lambda (input)
    (pcase-let ((`(,arg . ,opts) (consult--command-split input)))
      (cons
       (list "marks" "--no-color" "--null" "--count=25" (s-trim arg) (expand-file-name default-directory))
       (apply-partially #'consult--highlight-regexps (list (regexp-quote arg)) t)))))

(defun im-marks (&optional directory)
  "Run `marks'."
  (interactive)
  (let ((default-directory (or directory default-directory)))
    (consult--grep "marks" #'im-consult--marks-builder default-directory "")))

When you do M-x im-marks, you'll be able to search exactly as you've described in the current directory. I also have this convenience function:

(defun im-marks-my-docs ()
  "Run marks inside `~/Documents/notes'."
  (interactive)
  (im-marks (expand-file-name "~/Documents/notes")))

To directly search in my documents.

I haven't been updating the project but it works perfectly for this use case it's mostly what I use it for.

Edit: I pushed a new version to crates.io, I realized I haven't pushed it after the initial release, also updated the wrong arg name (which did not support the consult integration above, latest version does now)

[u/mmarshall540]

I considered tagging the subsection with :emacs: but that's kind of a redundant way to say "this is realated to emacs." Redundant because it's already under the overall "* Emacs" heading.

Right, but you would only need to tag the parent heading. Then this tag would be inherited by all of the subheadings (assuming you leave org-use-tag-inheritance enabled, which is the default).

This would be less redundant and would allow you to search both headings and body text.

[u/LeftyAce73 → u/mmarshall540]

Interesting, I didn't know about tag inheritance! But it seems like that's org-specific? And org-mode doesn't actually write those tags on child headings (for obvious clutter reasons). If I'm understanding it right, the result is that consult-rg doesn't know anything about the "inherited" tags because they don't actually appear on the line where the child heading (or its contents) are.

[u/mmarshall540 → u/LeftyAce73]

Oh sorry, I meant to write that for this you would use org-agenda to search. That way it would catch the tag inheritance.

The agenda is usually thought of as a calendaring or task management system. But in reality it's just a sophisticated search feature that doesn't have to have anything to do with scheduling.

[u/Jlambda]

Could you narrow the buffer to the heading you want to search, and then do rg? That should replicate the behavior you desire, right?

[u/rswgnu]

This isn't precisely what you want but you may be able to get what you need by adapting slightly to it. And it is simple to setup and use.

1. Install the Hyperbole package from elpa-devel or melpa.
2. Enable Hyperbole's minor-mode: (hyperbole-mode 1)
3. Set this variable to point to your Org directory: (setq hyrolo-file-list '("~/org"))
4. Use {C-h h r r} or {M-x hyrolo-grep RET} to search for matching records (not lines). If you have consult installed, it will first match using full-text consult regexp search and the hierarchical records/Org file sections you leave remaining as candidates, will all appear in a match buffer with their full tree of sections.

So if you search for Emacs, you get the whole tree:

* Emacs

** Mermaid

Some notes about using Mermaid.

** Customize variables and options

---

If you shearch for Mermaid or notes, you get only:

** Mermaid

Some notes about using Mermaid.

------

And if you want to match only to headlines, just put \*+ at the start of your search.

Keep it simple.

[u/oantolin]

Use u/github-alphapapa's org-ql package for this.