Keeping it local
by Matt Marshall on 2023-01-04 | License Permalink
I’m writing this on a machine with no wifi, because my broadband connection fell down about an hour ago and my service providers are not picking up the phone. I can tether using my phone’s data connection, but that’s only got about 2GB of allowance so I try to preserve it so that it doesn’t get eaten up automatically by data-hungry daemons such as Nextcloud. Besides, the majority of my computing takes place offline these days anyway.
This post explores the ways I’ve reduced my reliance on an active internet connection to get my daily computing done (both work and play) as well as the areas I’ve still got some work to do.
Projects and Work
Despite the growing integration of The Cloud™ in professional and personal work, I have managed to build a fairly robust system of getting things done using local tools and local files.
I don’t do much work at all on my smartphone; it’s largely for web browsing and listening to audio while moving about. The one exception is that it sometimes functions as a better calendar viewer than anything I’ve been able to achieve on the desktop!
Personal organisation and management
I’m really happy that I’ve got pretty much all of my personal organisation and management tools set up to run locally. The one glaring exception to all this my work calendar, which sadly I do need to check via Google Calendar in a browser or via an iCal feed in a phone app.
For personal information management, I have an entire folder of plaintext that I treat as a plaintext planner. It’s mostly markdown files with a few
.csv files thrown in for good measure. I wrote an entire post on it, which is a much better overview of how I use it than what I could produce in summary here.
The other main area of my life I manage locally is passwords. I’ve never used an externally hosted password management service such as LastPass and it appears that I am lucky. I manage my passwords using some CLI software called pass. It effectively allows me to create/remove/modify passwords stored locally and encrypt them using my PGP key. The CLI can read the password directly into my clipboard so I can enter it quickly and allows me to store some other information in the same file providing I don’t store it on the first line. It even has built-in git support, so I keep an offline backup on a Gitlab repo (the passwords are encrypted but the repo is still private) and there is actually an Android application. I don’t need to use passwords for logging into much on mobile devices, but I keep the app around because it pays to have another local copy of the repo somewhere just in case.
As noted, the main glaringly obvious hole in my locally-hosted paradise is that of my work calendar. My work calendar is hosted by Google Calendar and is pretty hefty, with some recurring events from 2018 still being used well into 2022 and showing no signs of stopping in 2023. Unfortunately it’s common practice for people to invite me to calls / meetings later in the same day. This means that if I want an offline solution, I need regular syncing. I tried several solutions including using the weighty graphical tool gnome-calendar, and even writing my own python script to download and parse the private
.ical feed from Google and present relevant details back to me in the terminal. Unfortunately the iCal file is too large for a script to parse quickly and, as noted, there are events in there that actually begin in 2018 and recur regularly. This means it’s difficult to automatically trim-down the file for use locally. I am still holding out hope for a CLI to iCal feeds or even just Google Calendar, but I do not have the skill to write it. The best solution I have come up with thusfar is to either occasionally view my calendar in the browser and also add it to my phone’s calendar so that I can get notified of new events if I’m away from the machine (ie making lunch early, or taking a break from the screen to read a book on the sofa). Please get in touch with your solutions if you’ve made more progress than I!
One other piece of personal organisation that I struggle to keep fully offline is my contacts. I generally manage these on my phone with the help of my Nextcloud instance and Dav×5. It does mean that at any given time there are at least three copies of contacts on local devices, but they’re not immediately local to my main desktop/laptop machines. This isn’t a priority for me at the moment but is something that flitters across my mind occasionally.
I write a lot. Thankfully, I enjoy writing. Historically, I’ve done a lot of different types of writing, but my toolkit generally remains the same. Most projects get a git repository which remains local to my machine and is backed up on a git instance such as Gitlab.
I have separate project folders for academic writing and my creative writing (when I say this I mean fiction, although I will concede that one can produce creative nonfiction!). It’s very rare that I need a dedicated project folder outside of these for writing projects although it does happen. One such case was my PhD thesis and it was separated out due to the amount of non-text assets it contained, which inflated the size of the repo considerably. In most cases, the writing project gets its own subfolder under the academic or creative writing folders. I generally write using Markdown these days although there are a few times I dip back into LaTeX for academia. In these folders I use
.gitignore to ignore an
/out folder, which allows me to use Pandoc to produce HTML or PDF output in that directory without committing the outputted files. This means I can produce finished publication-ready files in the same repository but avoids mixing non-source and source files in the commit history. Occasionally I’ll need to output to DOCX for sharing with the MS-Word-obsessed folks in the world of academia.
When I need to write documents for my day job I am generally writing proposals, research, summaries or reports. Unfortunately these often need to be done via some proprietary cloud software (Google Docs). These are more difficult to keep and produce locally, but if I am writing the first version of a report or document that generally means I can produce it in markdown on my machine. These reports generally live inside the appropriate “workbook” markdown file inside of my planner. I’ve then got an extension for Google Docs which converts the markdown to formatted text.
Programming and Data Work
I also do a bit of programming and some data work in both my day job and in my personal life. The programming isn’t particularly intense and usually consists of me writing bash scripts or python CLI tools to make some jobs in my life a little easier. For these, the standard “local git repo with online backup” model applies. My scripts are all in a single repo, whereas some tools which others may find interesting are separated out into their own project folder e.g. my Python implementation of todo.sh which has additional git features built in. For other work such as authoring schemas, or interacting with JSON data, I use commandline tools and text editors to get the job done inside the appropriate project repo, which is a local version of a git repo generally stored on a collaborator’s or my company’s github page.
For some tasks I need to run some analysis notebooks which were created using Google’s non-free fork of Jupyter notebooks known as Colab (I won’t link to this as it’s not FLOSS). If the dataset I’m working with is more than my little laptop can handle, I have to use the hosted Colab version out of Google Drive. If the dataset is more reasonable, I have Jupyter notebooks installed locally and, thankfully, most of our notebooks are available in git repos as
.ipynb files. It’s still using a web browser but if my internet goes kaput it means that I can still get the work done.
For some personal organisation such as household finances and such, I use some plaintext data files in CSV or YAML format and interact with these using scripts or text editors depending on what is most appropriate for the context.
I keep personal emails on my laptop machines. This is a welcome side-effect of the fact that I started transitioning to a more keyboard-and-terminal oriented workflow during 2021 and 2022. Part of this transition was swapping out Thunderbird for neomutt where possible and I made the setup a lot easier by using Luke Smith’s mutt-wizard. mutt-wizard not only sets up neomutt and points it to your IMAP or POP accounts, but also sets up your email to use isync (note, the binary is called
mbsync while the project is called isync). isync does the job of fetching my mail from the server while I use neomutt to read/write mail. I believe the sending of mail is also handled by a dedicated program, msmtp.
mutt-wizard was a revelation and made setting up this a lot easier, especially as I need to have multiple accounts configured. The only thing it’s lacking is a way to filter mail into various subfolders, but I’m working on that.
For my work email, it’s hosted via GMail which is no problem for neomutt and mutt-wizard but I receive a metric tonne of email every day from various sources and I simply couldn’t process it without a strong and flexible filtering system. Unfortunately this is one area I’ve had to concede to GUI apps (Thunderbird) and keeping the email on the server for now.
Entertainment and Media
So, primarily, I don’t subscribe to any streaming services for music. No spotify for me. I don’t like the weird neo-feudal attitude to streaming as it stops you owning anything and locks it behind paywalls. Right, rant out of the way; how do I listen to music?
Basically, the exact same way I’ve listened to it for about 16 years. I have a library of music files on my laptop(s) which are organsied via Artist/Album and then played through a music player either on my laptops or on my portable device. On my laptops this was iTunes when I was 14, then when I discovered GNU/Linux this was historically Rhythmbox but now I use mpd with ncmpcpp as a client. To listen to music on the move, such as walking through town or going for a run, I sideload the files onto my phone via USB. I’ve used a lot of music apps over the years but have recently landed on using VLC.
Most of my music library (around 80-90%) comes from CDs that I purchased between the ages of 14 and 20 or borrowed from friends. I originally ripped them via iTunes or later via ripperX on GNU/Linux. The files have very simply followed me around ever since. I had a brief flirtation with Google Play Music back during my fall to the dark side and when it allowed you to upload tracks and sync/pin them across devices, but that relationship ended when I got anxious about not owning my music collection and I retrieved my music collection back to my local devices (I later found that I’d also backed it up). I also did some pruning of my music collection at the height of my minimalism but, by-and-large, my music collection has remained intact since I began building it as a teenager with an iPod Classic.
I add to my library these days by exploring/discovering an artist and listening to a few of their tracks. This usually does involve an internet connection, admittedly. I’ve read a few places that the “album-era” is over and replaced by playlists and singles, but I grew up towards the end of the album era and I still think of albums as the basic unit of sitting and listening to music. This basically means that when I’m huntuing for new music, I will be generally looking for fan-curated lists of “top” albums by an artist. I’ll see if my peers have an album to listen to and rip or if that avenue isn’t open to me I’ll find a non-amazon vendor selling digital copies of the album DRM-free for me to download. If I’m familiar with the artist already I’ll skip the majority of the research although I will occasionally do a sense check to weed out any albums that are well-known to be “duds” by artists I otherwise enjoy. For albums I buy physical copies of, once I’m happy that the audio files are backed up appropriately I’ll give the CD to a friend as a gift or donate it to the local library.
I’m quite strict with my choice of digital format; it’s got to be ogg vorbis. I know FLAC is meant to be king but I’ve simply not got that amount of space on my hard drives, and it makes copying the files to a device an utter pain. If I can’t download or rip the files into ogg after I purchase them, I use dir2ogg as my converter.
Once I’ve got my ogg files lines up, I move them into an appropriate folder (Artist/Album) inside of my Music folder. These days I manage their ID3 tags via id3v2, or if they’re being really awkward and not responding I use easytag. It’s a GUI app, but it’s pretty fast. I’m not that fussed about album art these days since I use a terminal player with mpd, but I used to be pretty on the ball with this as the iTunes interface lent itself to showing album art.
Sometimes, I don’t want to listen to an album. I want to listen to some ambience or some Lo-Fi beats. I used to just fire up a browser and head to YouTube to open that famous Lo-Fi Girl/Chilled Cow stream (note: the link opens a link to the Lo-Fi girl stream on an Invidious instance to avoid opening the YouTube site). When I wanted to cut down on my browser usage, I simply pointed mpv at the URL because it can stream video content. When I wanted to ensure that I still had relaxing music to listen to offline, I learned that lofigirl.com actually release a bunch of albums by various creators for free. So every few weeks or months, I download a bunch of albums from there and add them to my local music library. To group them together, I just tag them all with the artist Lo-Fi Beats and ensure that the album and song titles reflect the creators. At present I have 11 hours of Lo-Fi to play on repeat or randomised if my network ever goes down.
My music collection is integrated into a regular backup regime for my local data, but I have a dedicated sync for it on my Nextcloud instance for off-site redundancy.
OK, so grabbing podcasts does require an internet connection. That’s kinda what the internet is for and it’s very good at it. But I love the interaction that podcasts inherently afford; being that you download them and keep them for later.
Maybe in 2023 most people are streaming their podcasts but I’ve never understood this at all. I originally listened to podcasts via my old iPod Classic, where I downloaded them via iTunes and synced them to the device. These days, I rarely listen to podcasts via anything other than my phone. For me they’ve always lent themselves to portable devices which have historically either not had an internet connection or had a hard download limit – and I’ve always erred on the side of requiring less data. I only recently upgraded from a 250MB data limit to a 2GB one!
So, my podcatcher of choice is AntennaPod. I subscribe to a bunch of feeds and then create a backlog of shows to watch. Usually I download about 3 hours of shows at a time when I have wi-fi access and then delete them as I go. I might top up this occasionally before I reach the end of the queue. For me, the interaction of “download now and listen at leisure” is just utterly perfect and to be honest has influenced the way I see how digital media should be.
I have been stung a few times by podcast feeds either disappearing entirely or expiring links to shows I’ve enjoyed and want to re-download. For shows I particularly enjoy, I use a desktop podcatcher known as gPodder (Side note; if there’s a comparable TUI podcatcher I don’t know about please let me know!) to download the feed en masse and back it up to an archive on a USB hard drive. When I want to listen to a show again, I generally tidy up the ID3 tags a bit to create an “album” and load it back onto my phone via USB where I play it using an audiobook application known as Voice. I’ve found audiobook apps lend themselves to podcasts a bit better than music players because they keep the playback position between listens as an inherent affordance of what they are.
Movies and TV
Similar to my music library, I store my video media locally. Well, actually it gets transferred to a USB hard drive pretty much straight away due to the space it takes up. But it’s still local, and I don’t subscribe to any streaming services.
This practice started because watching movies together was a family bonding activity for us when I was growing up. When DVDs had their hey-day and dropped dramatically in price, my mother managed to amass a truly astounding DVD library. When I got into computing and wanted to watch movies on my own I learned how to rip the DVDs to create some back-ups and to integrate them fully into my computing workflow. These files have stayed with me for over 15 years. The resolution isn’t great (especially on the early rips which were for the iPod Classic screens!) but they’re serviceable. The files have been joined by their more contemporary peers and have migrated across somewhere between 8 and 10 drives over the years.
We don’t have a TV at the moment, and likely won’t get one again. We mostly watch movies on my monitor (we have a cosy setup with a sofa facing the desk) and my partner is now pretty fluent in the keyboard commands for operating the nnn file browser and the mpv video player. I don’t watch TV or movies on the move, but if I ever go on a long-haul flight or train journey and need entertainment I’ll likely have my laptop and my drives. In an odd situation where I don’t want to rely on my laptop, I can sideload the files onto a phone or tablet via USB and play them locally via an app.
It’s 2023 now and there is some cracking video produced on the internet. I don’t subscribe to any streaming services, so I do miss some content there, but I do have a set of YouTube and PeerTube channels that I enjoy. Miraculously, I do manage to watch these offline most of the time, too. It’s almost the same process as grabbing podcasts.
If you read Esperanto then you can check out my post here on how to use an RSS reader, mpv, and feh to basically view most social media including YouTube and TikTok without ever leaving the terminal. However, in short I use youtube-dl to nab most YouTube videos from channels I “subscribe” to. It does download pretty slowly, but that means I can start downloading them straight away in the morning and have some viewable videos by the time I want to watch something (either at lunch or after I finish work). youtube-dl also works on non-YouTube websites so I have my download script wired to several RSS feeds such as some PeerTube channels.
For videos from sources I don’t subscribe to, such as when I need to find something on YouTube or PeerTube but don’t know who/what I’ll find; I do have to concede that this is an online task and I can’t really keep a local copy of the entire internet.
Photographs and home video
I don’t take a lot of photographs myself but over the years I’ve amassed a fair collection of memories in photograph form. When I have photographs taken either via a digital camera or my phone I just plug them into my laptop to transfer the files there. I use Shotwell which is a GUI application but is also incredibly useful at auto-sorting files based on EXIF metadata so that I’ve got everything organised by date (which is more useful to me than abstract collections). It’s also got a very handy import function for when a USB mass storage device like a digital camera or phone is plugged in. The same applies for home videos taken via the cameras/phones as well.
If others send me photos of myself (ie if we’ve all been on the same trip), I save them to my device’s memory and import them from there as normal. Or sometimes if we’re back at our place for dinner after a hike, I’ll nab the files directly off of my friend’s devices.
As my photos are stored in my home folder under Pictures, they’re integrated into my regular back-up regime as well as having a dedicated sync connection to my off-site backup via Nextcloud (see the Backups section).
If you’ve made it this far into the post, this model won’t be surprising to you. My books library is a mixture of paper and ebooks, and all of my ebook files are stored locally.
I read books on my e-reader which is an old 2011-era Kindle which has had the wi-fi access turned off since the day I got it. I sideload books onto it via a USB connection, and my library is managed by Calibre. I acquire the books themselves from a variety of sources, although these can be mostly summed up as “Project Gutenberg and a variety of non-Amazon book stores such as eBook store, Retbutiko, or Smashwords”. I also use my academic credentials to occasionally get books from e.g. Springer. I use more than these and really the only criteria I have are that they aren’t amazon and don’t enforce DRM on my books. I am fine with them watermarking a page with my name or an id number (I can always edit them out…) but I don’t want any physical restrictions on how I read a file I’ve paid for.
I have a strong preference for the EPub format, but Calibre’s conversion tools mean I don’t mind the odd stray format. I mostly view PDFs via my laptop, but occasionally sideload them manually onto my tablet when the mood strikes (it rarely does).
My entire calibre library is integrated into my back-ups routine, but I also take special pains to produce extra back-ups because of the amount of money and effort I put into acquiring and reading books. For the additional backups, I use calibre’s export function to produce some archive files which I put into a folder and back up to a dedicated USB pen as well as an SD card or two.
With so much of my computing taking place locally, backups become much more important. One of the reasons I try to reduce my reliance on an active internet connection is to reduce the fragility of my setup. The logic there is that by reducing the impact of a lack of internet on my work, the more able I am to continue regardless of connection. By the same token, I need to account for fragility in the machine itself – technology has an awful habit of failing at the least convenient moments.
Backups are therefore an important part of managing this fragility. If my machine fails that’s one thing but if it takes my work and data with it then that’s quite another.
Backups are one area I know I need to be better at, but I have a just-good-enough setup at the moment which keeps me going. In an ideal world, I’d follow The Tao of Backup to the letter (minus, of course, the commercial software that it advertises). For now, I’ve drawn lessons from it where possible.
My first line of backup is really some external hard drives. My TV and Movies and podcasts are all stored on an external drive by default, and I admit that this is an area I don’t diligently backup very much. My reckoning is that this is less “mission-critical” than my Documents, Books, Photos, and Music. But in the future I do want to ensure that this drive is mirrored somewhere else. For the rest, I semi-regularly (about once a month) backup my entire home folder (including configuration files) to a USB hard drive that is dedicated to the task. The back-up is dated, and there are multiple copies of it with older copies being compressed. The drive I use for backup explicitly has a robust case, although it doesn’t get transported much either. In addition to the collection of versioned backups on the USB hard drive, I have a USB flash storage (pen or thumb drive, pick your vernacular) which I use to create another copy although this just stores the current version and is overwritten each time.
The tools I use for this are pretty much standard GNU/Linux fare; rsync does the copying because it’s fast and scriptable and
tar and gzip do the compression.
My second line of backup is Nextcloud, which does require an internet connection and acts as an offsite backup. I don’t sync an entire “Nextcloud” folder, but I set up dedicated sync connections for each my Documents, Music, and Photos folders. I used to sync my Calibre library folder as well, but learned the unfortunate way that Calibre doesn’t like this and I had a few books corrupted (which I then had to reacquire). To back up Calibre to Nextcloud, I manually create a Calibre backup/export, date and compress it, and upload this to a folder on Nextcloud.
A parallel second line of backup, specifically for project files, is Gitlab. I have a Gitlab account which I use to store online remotes of my git projects such as my planner, blog, etc. in private repos.
People who work with backups professionally will probably be screaming at my lack of diligence and robustness, and this is an area I want to work on in 2023. To start with, I want to begin automating as much of the backup process as possible. At the moment I’m thinking of a dedicated desktop USB drive which is always plugged in and therefore can be used to automate a daily, weekly, and monthly backup of my home drive (managed via rsync), and the existing drives can be used to provide additional backups to this. I also want to set up a Raspberry Pi home server which acts as another backup via syncthing, and also acts as another remote for my git repos. I also want to reduce my reliance on an internet connection for my offsite backups. Some reading around the areas suggests that Blu-ray discs stored at a friend or relative’s home might be a good call.
This post discussed some of the ways I’ve managed to reduce my reliance on an active internet connection in order to do my daily computing for both work and play. I try to keep most of my work and Documents off of cloud services and on my devices, and I don’t rely on any streaming services for my media access. I have a backup system which mostly involves manually backing up to USB drives via rsync and I treat Nextcloud and Gitlab as offsite backups, although these do require an internet connection.
In the future my efforts will be concentrated on setting up a robust, automated, backup system for my work and media files. I doubt I’ll ever achieve perfection, but I can always strive for more robustness. Hopefully when the day comes that I need my back-ups, my efforts will have paid off.