Skip to content

Blog has migrated..

Scaling the humans (Elf-vengers, Assemble!)

Yesterday's blog post was heavy on the technical details re scaling our infrastructure for growth. Ironically, after bragging about how we're "ready" for growth, today we had an an issue affecting multiple users, which looks like it was caused by utilization / saturation 🤦

However, I can't confirm this, because.. the latest Grafana / Kube Prometheus Stack update broke graphing, and we've been flying blind for the last few days while I try to fix it while also adding new apps, welcoming new users, and handling routine support!

So the real problem here is the human (me) isn't scaling under load, which is why I've assembled you all to present my new initiative...

Scaling the gigabytes

We saw some stability issues earlier this week, as increased load impacted our ceph cluster, which provides the backend to the application config folders, as well as to ElfStorage.

It turned out that the 1Gbps nodes which run our SSD-backed config storage were also running the Ceph metadata servers, whose job it is to co-ordinate your view of the filesystems of your volumes. The combination of these two roles (storage and metadata) was saturating the 1Gbps NICs, causing slowdowns and the occasional corruption as the fault cascaded.

In parallel, all the fun we've been having with Real-Debrid streaming was impacting our app nodes, in some cases creating so much incoming traffic that the nodes were unable to respond timeously to communications with ceph, again resulting in slowdows and corruption.

Here's are a few recent changes we've made to address growth:

Autoscan Unbundled

Last month, we added Autoscan for free, bundled with Plex, Jellyfin or Emby.

With the Jan 2024 influx of "infinite streaming" users, Autoscan has taken a front seat, since it's key to updating your streamer libraries when new content is added to your remote library.

A hiccup has emerged re our deployment strategy though - it turns out if you have both Plex and Jellyfin, then you'll get two copies of autoscan, and they'll fight with each other for access to their SQLite database 🤦‍♂️

The workaround here has been simply to refactor Autoscan into its own product, and make it available in the store (for free).

TL;DR : If you lost Autoscan and you want it back, add it to your subscription via the link above, and then "renew now" your subscription to force the store to update it immediately.

Mooar debrid toys

It's too soon to tell whether it's "just a passing phase", but the "infinite" streaming app bundles seem to be a popular alternative to traditional torrenting / usenet. It's a good fit for us, since (a) the apps can be complicated to interconnect, (and automating complexity is our jam), and (b) they rely primarily on compute and network resources, which are cheap and easily expandable, rather than storage, which is a stone-age PITA 🪨.

So for your instant-streaming enjoyment, I present the following "instant streaming" guides:

We've also got an alldebrid bundle ready for a trial, if that's your flavor.

Read more for capacity expansion, and changes to TDarr and WebDAV access to meet resourcing constraints...

Beware, here be bugz

After yesterday's plex-debrid business, Discord (and the ticket channels) were surprisingly quiet today. Either the hiccups are gone and users are happily streaming, or they've given up and gone outside with the rest of r/realdebrid!

One minor improvement to note - you can now do a once-off top-up of ElfBuckz, without it turning into a monthly subscription.

We did find a few bugs today, which I'll list below, in order of criticality:

  • in new-app
  • 4 min read

Turn-key solution to Plex-Debrid streaming

Today we welcome plex-debrid, another piece of the puzzle for simple Plex debrid-based streaming. Plex-debrid lets you use your Plex watchlist as a "watchlist" (haha), scraping public trackers for the debrid-cached releases, and adding them to your Real-Debrid. It then triggers a rescan of the Plex library, so that in theory, the media is available to stream within 10-20s!

Plex-Debrid "Infinite" streaming

This is a beginner-friendly solution which avoids dealing with torrents, trackers, ratios, VPNs, and storage limits. The trade-off, of course, is that you don't own the media - it's streamed to you from your debrid service, for as long as you have a valid subscription.

Here's a little diagram showing how simple it can be:

flowchart TD
    K[Plex] --> |Stream|Z[User]
    Z[User] --> |Add to watchlist|K[Plex]
    K[Plex] --> |Watchlist|G[plex_debrid]
    G[plex_debrid] --> |Scan on download|K[Plex]
    G[plex_debrid] <--> |Find and add torrent|H[real-debrid]
    H --> I[Zurg]
    I --> |/storage/real-debrid/...|J[rclone mount]
    K[Plex] <--> |Library|J

And here's a demo from the plex_debrid repo..

Go behind the scenes with Kubernetes Dashboard

As you'll no doubt have read, ElfHosted is not a traditional seedbox, VPS, or shared hosting provider, but rather something new - we run the apps you choose, on a selection of appropriate hosts, in a highly-available and fault-tolerant way, using Kubernetes. And we open-source it all.

Kubernetes

What's the downside?

This distinction has many advantages (automation, scalability, resilience...), and some complex implications - for one, your apps don't necessarily run on the same hardware, so it may be that Radarr, Sonarr, and qBittorrent are all running on different hosts with different storage and processor capabilities. Our streamers (Plex, Jellyfin, Emby), for example, run on hosts with Intel Quicksync support, for hardware transcoding support.

We can't (easily) provide TCP-based access for services like SSH or FTP, which is why we rely on rlcone and WebDAV for data migrations, and we can't provide "root access" because each app runs on an as-locked-down-as-possible container, dispersed across our cluster.

One previously negative implication has been (until now...) that it's not simple to see app's real-time CPU / memory usage, or to watch logs in realtime. Certainly, these are available at a cluster-admin level, but without complicated authentication integrations, it's not possible to provide a user with visibility into their own resources, without exposing the privacy of other users.

A sexy solution!

We now have a solution I'm particularly proud of, because it exposes our Kubernetes environment on a per-user basis, in a way that's safe, while still providing all the cool visibility.. which I'm happy to now be able to share with you...

Geek out with custom domain names / branding for Jellyseer / Overseerr

Do you 1⃣ love showing off your Overseerr / Jellyseerr to the family and friends you share your media with?

Do you 2⃣ also love owning your own domain name, like iamacouchpotato.com?

Now, you can combine these two passions with custom domain names for Overseerr and Jellyseerr! 🏆

Setup custom domain for Jellyseer / Overseerr

Create a custom CNAME on your domain name (noobflix.iamacouchpotato.com, for example), pointed to the FQDN of your elfhosted service (i.e. noobflix-overseerr.elfhosted.com), and the pick up a custom domain addon for Overseerr or Jellyseerr.

Plug your CNAME in, push the button, and within a few minutes we'll have generated the appropriate SSL certificate for your fancy new https://noobflix.iamacouchpotato.com instance!

Read on for details re changing the default logos too!

Welcome to 2024! 🎉

In today's blog post, the first of 2024, we discuss:

  • How SABnzbd's queue in the 1TB /tmp folder now persists
  • How RDTClient gets some VIP network connectivity
  • How the volume provisioning bug was actually solved