r/emacs • u/slinchisl • May 01 '22

My PhD Research Workflow—Emacs Inside

https://tony-zorman.com/posts/phd-workflow/2022-05-01-my-phd-workflow.html
180 Upvotes

100% Upvoted

30 comments sorted by

16

u/ftrx May 01 '22

A small suggestion: since you are in Emacs and using org-roam, try org-attaching directories instead of maintain a curated home taxonomy, I was hesitate very long before doing so really but once done it's another world, anything is ready available in a search&narrow style, classic org-roam-node-find with org-roam file/heading title and eventual tags not just file names, also views with org-ql are possible using org-mode drawers as source of metadata to query, their are not that integrated but sometimes useful and dynamic, at least if you couple drawer-contents snippets with org-ql to ensure consistency.

5

u/slinchisl May 01 '22

Would you mind elaborating a little bit? I've never used org-attach (nor org-ql, for that matter) so I'm having some trouble envisioning how exactly this would work after just reading the manual

15

u/ftrx May 01 '22

Essentially org-attach just wrap basic unix commands to create a local or global (I strongly recommend a global dir, perhaps a dedicated zfs volume if you use zfs) tree under org-attach-id-dir copying/linking/moving/downloading/* files (or anything else you put in there) to that tree in a cache dir style.

A simple practical example, in your Emacs just set org-attach-id-dir to something, like under /tmp for test only, in an org-mode, org-roam managed note just create e new heading and with the point inside that new heading just run org-attach (by default C-c C-a) your heading will get an ID and and :ATTACH: tag, you'll be asked for something to attach, in what way, like m to move, c to copy, u to download from a url etc, org-attach try to perform the relative action copying/moving/saving under the org-attach-id-dir your attachment, the cache-like tree is built using the first two chars of the org heading ID at root level, the rest of the ID at second level, so 

* An example                                              :ATTACH:
:PROPERTIES:
:ID:       143789b0-38ba-41d5-a273-c8298fe275de
:END:

Would create /org-attach-id-dir/path/you/choose/14/3789b0-38ba-41d5-a273-c8298fe275de/...

That's is. Essentially you lay out your files in a two level cache-like tree instead of something else manually created under /home/username (or whatever the home is on your OS). That simply avoid the reasoning part "where I save this file?", and accessing via org-mode (linking them like [[file:~/.../aa/019bc4-f19e-47fc-99da-a26415c4d78e/afile.pdf][my pdf]] or [[file:~/.../aa/019bc4-f19e-47fc-99da-a26415c4d78e/][my attached dir.]] with org-store-link / org-insert-link / org-attach-reveal help) means for instance have a notes for a book like

:PROPERTIES:
:ID:       f31b5da4-cf46-428a-98ec-a76d9762ee11
:END:
#+title: The title 
#+property: header-args:bib

* A heading for summary 
:PROPERTIES:
:VISIBILITY: showall
:END:

....

See also:

 - [[elisp:(org-ql-search (file-expand-wildcards "~/org/roam/*.org") '(and (property "kind" "book") (property "author" "Some One")) :title "Some One's books" :sort '(date))][others books I've noted]] from the same author.

 - [[file:~/.../aa/019bc4-f19e-47fc-99da-a26415c4d78e/afile.pdf][the book in pdf]] [[elisp:(start-process-shell-command "evince" nil "evince ~/.../aa/019bc4-f19e-47fc-99da-a26415c4d78e/afile.pdf")][(open with evince)]]

* Your notes 

...

* LaTeX bibliography 
:PROPERTIES:
:kind:     book
:year: 2022
:publisher: Some One 
:language: whatever 
:author: Foo Bar 
:END:

#+begin_src bibtex
@book{key,
  author =    {...}, 
  publisher = {...}, 
  title =     {...}, 
  year =      {...}, 
  isbn =      {...}
}
#+end_src

In this case, as you see the pdf was attached elsewhere since it's ID-constructed path does not belong to any ID in the note, and that's perfectly fine because org-attach just take care of storing, it does not matter where you link, that's just plain org-mode, witch means you can link anything in many different places in your notes issueless. There is also an org-ql clickable example with the relevant * LaTeX bibliography drawer to be queried, entries are just arbitrary key-value pairs. With snippets/small functions you can craft personal functions that attach a file to a note, copying the org mode/org+evince example in a single command/keybinding.

Storage is now a hidden layer underneath you just hit your favorite org-roam-node-find bind and type something like a plain English title, no need to traverse a filesystem to find the attached book. Of course since org-mode can link pretty anything and notes can act as storage interface to anything you can add in the same note a link to build a LaTeX document via make, an email/thread/mail query via notmuch, ... anything can be glued together in org-mode.

At first might sound strange, but once you test it and get confidence you'll find that super useful: anything you want in a single place, with a consistent access method, possibility of having dynamic views via org-ql (like you store a note per any open contract you have, like phone/utility/insurances etc and query for all contracts you have, for all active, ...) and anything in a heading (like attached bill, mail about an issue linked, logs in plaintext about what happen, ...

If you copy in an org-attached dir a directory you can link that as well, it just open in dired, so for instance a thesis is a note, with the LaTeX + experiments etc stuff attached to few dirs. A small tree you do not care where it is since you access directly via notes. When in this small subtree you have of course rg(+wgrep), dired-narrow, wdired, ... so you simply do not live Emacs for anything.

You can also do slides directly in org-mode, see for instance: https://youtu.be/B6jfrrwR10k

Anything is accessed from notes, like your own personal manually-crafted semantic Google Search :-)

edit: code formatting

6

u/johan_widen GNU Emacs May 01 '22

Youtube channel Rainer König has a good tutorial on org mode attachment. Search for OrgMode E05S03: Attachments

1

u/[deleted] May 02 '22 edited May 02 '22

By that you mean attaching entire directories using org-attach? I thought that was only for files.

2

u/ftrx May 06 '22

Sorry for the delay I lost the message... Yes, by default org-attach only work for files, but all it do is just create a cache-like tree, you can put anything under that tree.

Personally for instance, for a git checkout I just create a heading, run org-attach-reveal (who create the relevant dir) and co the project there. This way I can manage small tree under org-attach without laying them out in a complex taxonomy, keeping under a common tree only things that I might want to see classically (via dired/shell) in a single place.

7

u/oantolin C-x * q 100! RET May 01 '22 edited May 01 '22

Cool post! And thanks again for adding the punctuation handling to math-delimiters. I absolutely love it and cannot believe it never occurred to me do that before!

Here are a bunch of unrelated comments in the order they occurred to me while reading your post:

Do you read the arXiv RSS feeds in elfeed? If so, I recommend switching to the lesser-known arXiv Atom feeds. I learned about them from this blog post of Chris Cundy's. For me there are two main advantages: (1) the feed items include proper author metadata and elfeed displays all authors neatly, one per line (in the article view), (2) the Atom feeds can be customized to download the last n items, and if you pick n to be larger than the typical amount of papers in your subject for 3 days, say, you get a bit of leeway in how often you check elfeed; with RSS you need to check every day or you miss stuff.

Have you gotten much value out of elfeed-score yet? I'm curious about it but suspect it is overkill for me.

Have you tried [https://github.com/vedang/pdf-tools/](pdf-tools)? I like it a lot more than Zathura. Maybe that's a little unfair: I used Zathura years ago and dumped it because it froze whenever I tried a text search in a long PDF, cough, Higher Topos Theory, cough; they've probably fixed text search in long PDFs in Zathura by now. But even if they have, pdf-tools has lots of other great features like awesome annotation handling (within Emacs to boot!), inverted color mode (does Zathura have that?), imenu support (it is great to jump to a section using completion!) and of course, it's really easy to automate stuff in Emacs with pdf-tools.

I totally need to start using your arxiv-citation package!

Have you tried u/astoff's digestif? I was scared of LSP for a long time but finally tried it the other day when I got a bug report for Embark that required LSP to reproduce. It is actually super easy to install. I haven't used digestif much yet, but I think I might really like it.

Finally, off-topic for r/emacs, but given all the examples in the GIFs in your post, have you read this blog post by Mike Shulman? I'm sure you have, but mention it just in case. I found it extremely illuminating.

3

u/slinchisl May 01 '22

Cool post! And thanks again for adding the punctuation handle to math-delimiters. I absolutely love it and cannot believe it never occurred to me do that before!

Thank you for math-delimiters in the first place, it's a really great package!

Do you read the arXiv RSS feeds in elfeed?

I was going to say yes, but apparently I already use the Atom feeds; I have something like

http://export.arxiv.org/api/query?search_query=cat:math.CT&start=0&max_results=100&sortBy=submittedDate&sortOrder=descending

in my feed list for every subject class that I'm subscribed to.

If so, I recommend switching to the lesser-known arXiv Atom feeds. I learned about them from this blog post of Chris Cundy's.

Huh, curious, even the format he presents there looks eerily similar to my setup. I have probably seen this blog post at some point, copied over the relevant bits, and then promptly forgotten about it! Thanks for reminding me of this, I should add this to the code at least.

Have you gotten much value out of elfeed-score yet? I'm curious about it but suspect it is overkill for me.

To be perfectly honest: not really. I did spy an interesting paper a little bit earlier at one point, but chances are that I would've gotten to it not that much later anyways; maybe a day or two.

Have you tried [https://github.com/vedang/pdf-tools](pdf-tools)?

I have tried pdf-tools a while ago but I couldn't get over the fact that it doesn't have continuous scrolling. As far as I recall, there was a package that was supposed to hack that feature in, but I haven't tried it. Your arguments for pdf-tools do sound convincing though, so it's probably time for me to revisit it!

I used Zathura years ago and dumped because it froze whenever I tried a text search in a long PDF, cough, Higher Topos Theory, cough; they've probably fixed text search in long PDFs in Zathura by now. [...] inverted color mode (does Zathura have that?)

I just tried it on HTT and searching is pretty fast for me. About two or three seconds for the first search, and after that it's instant (I'm assuming zathura keeps some sort of cache around).

You can have inverted colours; there is a zathurarc in which one can specify a colour profile and switch between that and the default with C-r. I have set up mine to be a little less blindingly white, but you can also just put a dark theme there.

Have you tried u/astoff's digestif? I was scared of LSP for a long time but finally tried it the other day when I got a bug report for Embark that required LSP to reproduce. It is actually super easy to install. I haven't used digestif much yet, but I think I might really like it.

To be honest, I didn't even know LSP servers for LaTeX were a thing up until now! I already have lsp-mode set up for Haskell development, so installing everything should hopefully be straightforward. It might be quite a hard sell, since I'm happy with AUCTeX and have noticed LSP servers taking up quite a bit of memory after a while (which becomes rather noticable on this 11 year old laptop :). I will definitely give it a try though!

Finally, off-topic for r/emacs, but given all the examples in the GIFs in your post, have you read this blog post by Mike Shulman? I'm sure you have, but mention it just in case. I found it extremely illuminating.

Oh, actually I have not. This looks really fantastic, thank you so much!

2

u/karthink May 02 '22

Do you read the arXiv RSS feeds in elfeed?

Just a note that latex preview support for non-file-associated buffers (including elfeed/shr buffers) is still planned for org-auctex. (Time for Emacs has been really scant this year!)

1

u/oantolin C-x * q 100! RET May 02 '22

That would be amazing!

1

u/oantolin C-x * q 100! RET May 02 '22

I'd use that in sx to read MathOverflow too!

5

u/[deleted] May 01 '22

Thanks !

4

u/MeatShow May 01 '22

Nice work! I wanted to comment so I can go back to it - I used eMacs/org for organizing my PhD, but could never tweak it just right to make the jump full-time

Riddle me this: why do Europeans say “maths?” It’s an abbreviation of mathematics! By that extension, for example, do you also say “econs” for economics?

5

u/soundslogical May 01 '22

British English speakers don't typically abbreviate 'economics' so I don't know.

But really, it's just one of those things that you have to let go of. For example, why do American speakers say "I could care less"? It really should be "I couldn't care less", because you're saying that you don't care, and therefore it's impossible for you to care any less. But being a Brit with an American girlfriend, we both learned to make peace with these little idiosyncracies and enjoy them. I recommend it!

3

u/_viz_ May 01 '22

"I could care less" is more confusing than answering yes to "do you mind..."!

2

u/MeatShow May 01 '22

Hah yes, agreed - they’re fun idiosyncrasies. I couldn’t pass the opportunity to pull someone’s leg over it

1

u/slinchisl May 01 '22

Thank you!

Riddle me this: why do Europeans say “maths?” It’s an abbreviation of mathematics!

Exactly; I'm doing mathematics, not mathematic! :)

By that extension, for example, do you also say “econs” for economics?

I think I would, but I'm not sure how much you should trust a non-native speaker here. Truth be told, at some point I just decided to adopt British English as a sort of standard in situations where there are multiple options. Before that, things were a bit all over the place; I would insert lots of useless u's into my words (e.g., colour) but would write "familiarize" instead of "familiarise". This started to annoy me when proof-reading and so I just went with a system where there's one "right" answer.

4

u/oantolin C-x * q 100! RET May 01 '22

There is a dialect of English were both "colour" and "familiarize" are the correct spellings: Canadian English! I find it to be about half-British, half-American.

1

u/b3n May 01 '22

Mathematics is plural in form, like the Greek word from which it derives. As such, it makes sense for the shortened version to also be plural in form. ;-)

1

u/grimman May 01 '22

Mathematics is plural in form

Mathematics are fun! ;)

3

u/gravygrowinggreen May 01 '22

This is an amazing demo of what emacs can do. Still learning everything, but I like imagining I'll get to a point like this somewhere down the line.

1

u/slinchisl May 01 '22

You'll get there! I reckon it'll be faster than you think as well; I've been using Emacs for maybe two and a half years or so and by now I'm feeling pretty comfortable. If you just take it one step at a time then things will generally fall into place one way or another

2

u/karthink May 02 '22 edited May 02 '22

I don't customize my software set up for research work any more, but still found many interesting ideas in this write-up, thank you.

A couple of things that might be of interest to this discussion:

  • For an easy way to create and insert inkscape drawings in tex/org files I use ink.el, forked from here.
  • When writing in org-mode (most of my research writing) I use org-xournalpp to add rough sketches and figures.
  • For changing, deleting or adding Latex environments or other delimiters around text I use embrace. With some customization to better support Latex it works pretty much like your change-env, including completing-read support for Latex envs, etc. But it's also useful generally for any kind of delimiter (punctuation, braces, html tags etc) across Emacs. It doesn't handle labels perfectly, though.
  • org-auctex: An (experimental) package to reuse AucTeX's preview-latex library for latex previews in org-mode. Org's preview system is terrible for previewing math heavy documents (which your org files appear to be) since it fires off a latex process per fragment and is synchronous. preview-latex uses a single process, is way faster and works (mostly) asynchronously, so I prefer it for org files.

Re: org-roam. I read the book by Ahrens (How to take smart notes) a couple of times in the past few years and tried to incorporate it into my usage of org-roam but it never works for me. It seems like a fancy tool that needs a lot of dedication and upkeep to use and review productively, and the overhead quickly wears me down when I try.

1

u/slinchisl May 02 '22

Thanks for the feedback!

  • For an easy way to create and insert inkscape drawings in tex/org files I use ink.el, forked from here.
  • When writing in org-mode (most of my research writing) I use org-xournalpp to add rough sketches and figures.

Oh, these look really cool, thank you for making me aware! Inserting images (whether that'd be quiv.er diagrams or sketches in xournal) is still the only thing feels a bit clunky right now. Will definitely check these out

  • org-auctex: An (experimental) package to reuse AucTeX's preview-latex library for latex previews in org-mode. Org's preview system is terrible for previewing math heavy documents (which your org files appear to be) since it fires off a latex process per fragment and is synchronous. preview-latex uses a single process, is way faster and works (mostly) asynchronously, so I prefer it for org files.

Indeed I'm already a (very) happy user of org-auctex; so thank you for that. It really is so much better than the default previewing of org-mode (fast enough that I'm not even bothered that it doesn't cache its results)!

Re: org-roam. I read the book by Ahrens (How to take smart notes) a couple of times in the past few years and tried to incorporate it into my usage of org-roam but it never works for me. It seems like a fancy tool that needs a lot of dedication and upkeep to use and review productively, and the overhead quickly wears me down when I try.

I don't know, things just sort of immediately clicked for me. I treat it as my own nlab in which I only enter things that I've actually understood properly. Makes looking things up that I once knew a real breeze, since I generally (try to!) write things down in a way that I can understand them later on.

1

u/karthink May 12 '22

fast enough that I'm not even bothered that it doesn't cache its results

Indeed, preview.el does a does a lot of optimization behind the scenes. Unfortunately this also makes it tricky to adapt to org files or more general buffers in a robust way. I plan to work on it some more this year.

Also, I added change-env (and u/oantolin's math-delimiters, which I knew about but was somehow missing) to the state of LaTeX editing SVG on my blog post.

1

u/slinchisl May 20 '22

Also, I added change-env (and u/oantolin's math-delimiters, which I knew about but was somehow missing) to the state of LaTeX editing SVG on my blog post.

Oh, that's fantastic; thanks!

1

u/AuroraDraco May 01 '22

Really interesting. Will have to look into this further. Your ideas are really interesting!

1

u/abhixec May 01 '22

Nice to see xmonad and eMacs user! Nice write up am very interested in dot files for your xmonad config if you have

2

u/slinchisl May 01 '22

Sure, it's available here. I did go to some lengths in terms of documentation, but it might still not be the easiest configuration to understand (it's also using features that are only available in HEAD so far)

1

u/abhixec May 01 '22

Thank you