Our platform is built on Hetzner auction server nodes, each of which has unlimited bandwidth, but is physically constrained to a 1Gbps NIC. As we've been bringing more subscribers on board, we've started to see occasional poor performance from the streamers (Plex, Jellyfin, Emby), most likely due to congestion on these 1Gbps links.
When you look at the graph below, the problem is immediately apparent (note the change when our maintenance window started):
So, today I added 5 more nodes to the cluster, to address two areas I felt were likely to become resource bottlenecks..
It turned out that due to changes in Radarr v4 compared to v5, our image wasn't passing testing, and so we didn't get the juicy new v22.214.171.12427 release a few weeks ago.
I was under the impression that v5 would support multiple resolutions for content, meaning a separate Radarr4K wouldn't be required anymore, but in the brief testing I've done so far, I've not been able to work out how to do this!
A longstanding issue we've been dealing with is that the Wordpress plugin we use to provide ElfBuckz doesn't include the ability to notify users when their balance is running low. So unless you manually keep track of your ElfBuckz balance, you're at risk of having your services stop when your ElfBuckz run out, without notice!
Since we recently fixed the deprovisioning bug, this has become even more important, so today I finally got my hands dirty and modified the plugin myself, adding the ability to notify users when their account balance goes low.
So.. you should now be alerted whenever your ElfBuckz balance drops below $1, which should give you plenty of time to arrange a topup (of course, once you subscribe to a top-up with the appropriate quantity, your ElfBuckz will "refill" every month, and you may never see these notifications)
How does this work? Like GitHub Pages, if you create a DNS CNAME record on your domain (mattermost.spankypants.com) and point it to your ElfHosted URL (spankypants-mattermost.elfhosted.com), we can request a LetsEncrypt certificate for that domain, validate it, and serve your app on that URL using our Traefik ingress controller.
Custom domains don't make sense for all apps, and I've not tested the impact on SSO (since auth cookies are domain-based), but as I add the option to apps, you'll find custom domains available at https://store.elfhosted.com/product-category/custom-domains/, and instructions added to the docs for the various apps.
Today we found a bug which has existed since our code was originally Seedplicity, which prevented the Notifiarr client UI from properly authenticating you, a necessary step towards configuring notifications for local services such as Radarr, Jellyfin, etc.
If you're unfamiliar with Notifiarr, it's worth a look - check this out:
I was trying to catch up on my 1080P movie backlog on my laptop recently, only to discover that Jellyfin would transcode the movie since the 5.1 audio track apparently won't work on modern Firefox on my modern Macbook (grr, why? )
Rather than just watching on my nVidia Shield (which plays everything I throw at it), I instead spent about 6 hours (I never even got to watch the movie!) adding Tdarr to our platform, with a mind to having a 2-channel audio track added to my existing media.
So, we now have Tdarr available! It works as advertised, although it turns out to be quite tricky to configure, and even trickier to make work with the Quick Sync Video hardware transcoding support in our 9th-gen Intel nodes.
A few peculiarities to our implementation:
While Tdarr can support multiple, distributed worker nodes, we just use one worker node, paired with the Tdarr server, for easy resource isolation, and because our transcoding "scratch" volume is actually a 200Gb ephemeral NVMe-backed disk, which only exists for as long as the pod does.
We don't actually want to significantly increase our CPU usage across the platform - rather, we want to leverage Tdarr for spare capacity on hardware-based transcoding. Each Tdarr pod has minimal CPU, and enough RAM to run the server, plus one hardware transcode, but will crash and restart if trying to do more than one transcode in parallel (because mooar transcodes requires mooar RAM)
I've pre-configured a working Tdarr setup, so you won't be starting from scratch when configuring it, and I've added more specifics to the app page.
Plex recently sent notices to users hosted on Hetzner IP ranges, indicating that from 12 Oct 2023, their hosting provider will be blocked.
Although the letter didn't mention Hetzner by name, it's pretty clear that they're the target, presumably due to their affordable (woo!) pricing, and the prevalence of "Plex Shares" services using it.
This presents an issue for ElfHosted, since we're using Hetzner for the same reasons (the price is nice).
Luckily (ironically) due to Hetzner's infamous zero tolerance of DMCA notices, we've already built out the infrastructure to add VPNs to pods, and allow users to "BYO VPN". This is how our torrent clients work - you've got to bring your own VPN credentials to play.
So, come 12 Oct, when Plex blocks Hetzner's IP ranges, it may be necessary to BYO VPN in order to keep using your Plex instance.
Today's new app is a fresh request from one of our fellow elfies.. Joplin is a 100% open-source , markdown-driven note-taking app, similar to Obsidian or Logseq. There are apps for Mac, Windows, and Linux, even a terminal app!
Joplin Server is the open-source version of the sync engine behind https://joplincloud.com, which takes you beyond simple apps, by enabling cross-device sync, sharing, publishing, etc.
Adding Joplin Server to your "Elf Stack" will let you utilize your ElfStorage (pass go, collect 100GB) for storing notes, attachments, etc.
The "publish note" feature is pretty - here's an example of what a published note looks like - it's published by joplin-server (so you can edit it with any client on any device), and it's auto-updated whenever the note is updated.
If you don't trust a seedy seedbox host with your super secret notes (and why should you?), you can enable E2E encryption in the client apps, which (like Seafile) means that all that's stored in ElfStorage is the encrypted blobs, and some metadata in the postgresql database.
I had feedback from a user today after a recent post about exposing various app APIs without SSO, pointing out that that SSO was applied a little inconsistently across our apps. For example, Jellyseerr was exposed publically (it has strong auth and is used to share media requests), but Overseerr (of which Jellyseer is a fork) was not!
This is now fixed
To clarify, here's how SSO is applied, based on the intended audience / purpose of an app:
Apps that only you (the user) would use, are behind SSO without any native auth, where possible. This includes download clients, apps without auth (openbooks), Arrs, etc (although these can be optionally exposed to a limited degree)
Apps which you might want to share (Plex, Navidrome, Kavita, Overseer, etc) and which provide their own trusted auth, are exposed without SSO, since their entire purpose is to share media / content with a wider audience than just yourself!