Organizing a Complex Directory for Emacs Org Mode and Deft
Deft allows you to search text files in a directory quickly on Emacs. It has an optional support for recursive search, which allows you to browse through a nested directory hierarchy. This post describes how to set up a complex repository for Deft that can be synchronized across multiple machines. It also explains an idea for improving the presentation of Deft listing files from multiple directories.
This post mainly consists of two parts: Synchronization and Configuring Deft:
- Synchronization discusses how to synchronize files in a Deft repository across multiple machines. I have adopted a hybrid method of using both Dropbox and Git, and I am describing its gist.
- An example repository structure presents how I am actually trying to organize my Deft repository based on this idea.
- Configuring Deft describes my setup for the Deft repository. It contains an example of customizing the presentation of Deft through a function advice. I hope it will be useful to you.
You probably need a way to synchronize files in your Deft repository across machines. Even if you usually use only one machine, you have to backup your contents to a separate location, and backup is often implemented by means of synchronization, so I will discuss synchronization methods in this section.
Dropbox vs. Git
There are some options for synchronizing text files between machines. The most popular ones among developers would be Dropbox and Git. Both options have their own pros and cons, so let’s compare those.
Dropbox is a file synchronization service.
- Files in Dropbox are automatically synchronized without human intervention.
- Its free plan doesn’t support history, and hence you can’t rollback changes to files.
Git is a revision control software.
- You can view the history of changes, and you can rollback files to a previous version.
- Even if files are simultaneously edited, their changes can be merged through manual resolving.
- It may require human intervention for operation. You have to make commits to the Git repository and run git push/pull for synchronization.
Using both Git and Dropbox
My solution is to use both Git and Dropbox.
My ~/org directory is basically a Git repository, but it also contains symbolic links to directories in Dropbox. By using this setup, some files can be tracked by Git, and other files are synchronized by Dropbox, while they coexist under the same root directory:
- Git repository (~/org)
- files: Tracked by Git
- files: Tracked by Git
- DirC (a symbolic link to a directory in Dropbox)
- files: Not tracked by Git, but synchronized by Dropbox
You must contain symbolic links inside a Git repository. It is impossible to maintain a Dropbox directory containing symbolic links. If a directory synchronized by Dropbox contains a symbolic link to another directory, the Dropbox daemon will copy its contents to other machines rather than reproduce symbolic links.
I suggest that you should use relative paths to refer to directories outside the repository so that you can deploy the repository to every machine regardless of the UNIX login name. If you use absolute paths, you may have to recreate symbolic links on machines that have different login names, which makes the setup a little more complicated.
Alternative solution: Using Resilio Sync for synchronizing the root repository
Resilio Sync is a file synchronization solution based on the BitTorrent protocol. You can use it to synchronize Org files across multiple machines in place of Dropbox.
Unlike Dropbox, Resilio Sync does not follow symbolic links. If a sync repository contains a symbolic link to a directory, neither files in the directory nor the symbolic link itself are synchronized by Resilio Sync. You can take advantage of this feature to manage a directory containing symbolic links to Git repositories:
- A directory tracked by Resilio Sync (~/org)
- DirA (a symbolic link to a Git repository)
- files: Tracked by Git in the separate repository
- DirB (a symbolic link to another Git repository)
- files: Tracked by Git in the separate repository
- DirC (a normal directory)
- files: Synchronized by Resilio Sync
- DirA (a symbolic link to a Git repository)
That is, instead of putting symbolic links to Dropbox directories in a Git repository, you can put symbolic links to Git repositories in a directory that is synchronized by Resilio Sync. The symbolic links are ignored by Resilio Sync, and files in the Git repositories are not copied to other machines. You have to generate those symbolic links on every machine though.
However, I abandoned this setup in favor of the “mega repository” way which I had already described. I didn’t like the idea of scattering Org files across multiple Git repositories. I also wanted to store some emacs lisp files in the repository, which should be managed by Git. Using a single mega Git repository allows you to track everything relevant under the same history, which is simpler and more consistent.
An example repository structure
I am currently using a directory hierarchy of the following structure. I am still under experiments to organize my Org files, so I am not confident with this setup at all:
~/blog: My Hugo blog source
~/Dropbox: My Dropbox repository
todo: A directory for org-agenda-files
seq: A directory for other org files
~/org: Git repository containing Org files
elisp: A directory containing Emacs Lisp files
archives: A directory for archive files
resources: A directory for org files for referencing purposes
- … (More directories and files)
Org files for everyday tasks are kept in
todo directory and automatically synchronized by Git, so you can work on them from multiple machines without manual Git operations. Archives, resources, and long-term goals are part of the Git repository, so you can review the history of those items.
resources contain Org files.
deft-directory variable is set to
~/org directory, so you can search all files in those directories using Deft.
blog-posts directory contains Markdown files for my blog posts including drafts, and you can search them as well using Deft.
Loading Emacs Lisp files from the repository
elisp directory contains Emacs Lisp code related to Org Mode. At first, it was part of my main dotfiles repository, but I moved it to this place, because its logic is closely coupled to the organization of my Org files. They constitute a whole system together, so they should be tracked in the same repository.
I am using Spacemacs. To load Emacs Lisp files from the directory,
~/.spacemacs.d/init.el should contain:
(setq my-org-directory "~/org") (setq my-org-elisp-directory (expand-file-name "elisp" my-org-directory)) (when (file-exists-p my-org-elisp-directory) (add-to-list 'load-path my-org-elisp-directory))
dotspacemacs/user-config should contain:
;; org layer configuration (let ((org-init-file (concat my-org-elisp-directory "/configure-org.el"))) (if (file-exists-p org-init-file) (load-file org-init-file) (error (concat org-init-file " does not exist!"))))
Now you can put configuration specific to Org Mode in
~/org/elisp/configure-org.el. After updating the repository using
git pull, you should reload Spacemacs with
SPC f e R or simply restart Emacs.
To use Deft with Org Mode on Spacemacs, you have to turn on
Options for Deft mode
I set options for Deft as follows. You have to put the code in
(setq my-org-directory "~/org") ;; deft layer configuration (setq deft-directory my-org-directory) (setq deft-extensions '("org" "md" "txt")) (setq deft-default-extension "org") (setq deft-recursive t) (setq deft-use-filename-as-title nil) (setq deft-use-filter-string-for-filename t) (setq deft-file-naming-rules '((nospace . "-")))
Most of the code above is obvious. Deft recursively searches text files from ~/org directory, and the default file type is
org. For documentation, press
<f1> v to run
describe-variable on an item below your cursor.
With this setup, you can quickly create a new file based on your input with
C-c C-n, but I actually rarely use this feature. Instead, I enter the title of a new file and press
C-RET, followed by a directory and a file name without an extension.
I also added custom keybindings to allow navigation in the deft buffer with
C-n/p. Put the following code in
(with-eval-after-load 'deft (define-key deft-mode-map (kbd "C-p") 'widget-backward) (define-key deft-mode-map (kbd "C-n") 'widget-forward) )
Advising a function to provide better titles
Deft allows you to configure to some extent how it displays file information in the deft buffer. By setting
deft-use-filename-as-title variable, you can choose whether Deft should display titles extracted from file contents, or display the relative file paths of files under your deft directory:
- If you set
nil, Deft will extract the document title from each file, but you will be unable to know where each file exists.
- If you set
t, you can display the relative paths of text files, but the file names may not be always descriptive enough, and it may take a little time for you to guess what a file is.
I was unsatisfied with both of these options, so I have defined a function to produce an entry title containing both a directory and a document title.
my-deft-title.el and put the following code:
(defun my-deft/strip-quotes (str) (cond ((string-match "\"\\(.+\\)\"" str) (match-string 1 str)) ((string-match "'\\(.+\\)'" str) (match-string 1 str)) (t str))) (defun my-deft/parse-title-from-front-matter-data (str) (if (string-match "^title: \\(.+\\)" str) (let* ((title-text (my-deft/strip-quotes (match-string 1 str))) (is-draft (string-match "^draft: true" str))) (concat (if is-draft "[DRAFT] " "") title-text)))) (defun my-deft/deft-file-relative-directory (filename) (file-name-directory (file-relative-name filename deft-directory))) (defun my-deft/title-prefix-from-file-name (filename) (let ((reldir (my-deft/deft-file-relative-directory filename))) (if reldir (concat (directory-file-name reldir) " > ")))) (defun my-deft/parse-title-with-directory-prepended (orig &rest args) (let ((str (nth 1 args)) (filename (car args))) (concat (my-deft/title-prefix-from-file-name filename) (let ((nondir (file-name-nondirectory filename))) (if (or (string-prefix-p "README" nondir) (string-suffix-p ".txt" filename)) nondir (if (string-prefix-p "---\n" str) (my-deft/parse-title-from-front-matter-data (car (split-string (substring str 4) "\n---\n"))) (apply orig args))))))) (provide 'my-deft-title)
You also have to put the following code in
(require 'my-deft-title) (advice-add 'deft-parse-title :around #'my-deft/parse-title-with-directory-prepended)
my-deft/parse-title-with-directory-prepended is a function to produce a title, implemented as an advice around
deft-parse-title function. Advice is a concept in Emacs Lisp: You can change the behavior of an existing function by adding an advice to it.
Each title produced by the function has a form of
DIRECTORY > DOCUMENT TITLE, where
DIRECTORY is the relative directory of a file and
DOCUMENT TITLE is a document title (if possible) extracted from the content.
The document title is produced from the filename and/or the content according to the following rules:
- If the filename is a
README.*, the title will be the filename itself. This will be enough, as its directory provides information on the context.
- If the filename has a
txtextension, the title will be the filename.
- If the file has a YAML front matter in the beginning denoted by
---, the title will be extracted from it. If the file has a draft status in the YAML front matter,
[DRAFT]tag will be prepended to the title.
- Otherwise, the default parsing function (
deft-parse-title) will be used. That is, the title will be the first line of the file, but extra characters in Org Mode metadata and Markdown headings are stripped out.
The following screenshot presents an example:
You can compare it with the screenshots of examples displaying only titles and only filenames. My setup is the most complicated, but it is the information-richest at the same time, which I think is crucial in a nested directory setup.
I feel comfortable with this setup. You can search text files in the repository by combining a directory name, part of a title, and part of file content as your query, and they are visible in the deft buffer, except for the content.
The following is a list of what I love with this Emacs Deft setup:
- Org files for everyday tasks in Dropbox are synchronized across machines without human intervention.
- Other Org files are tracked by Git, so they are safely backed up and you can review the history using
- Emacs Lisp code for configuring Org Mode is tracked by Git along with the actual repository contents, so your Emacs configuration is kept consistent with the org repository contents after reloading Spacemacs.
- Files in the Git repository, files in the directories in Dropbox, and blog posts can be searched from the single place. The list contains information on both the directory and the human-friendly title of each entry to help you pick the right item. This will make Emacs even more competent as a personal knowledge base.
This repository setup will be a fundamental for my workflow. I must maintain it to work properly. I also have to develop further configuration in order to get fully organized.