After yesterday's plex-debrid business, Discord (and the ticket channels) were surprisingly quiet today. Either the hiccups are gone and users are happily streaming, or they've given up and gone outside with the rest of r/realdebrid!
One minor improvement to note - you can now do a once-off top-up of ElfBuckz, without it turning into a monthly subscription.
We did find a few bugs today, which I'll list below, in order of criticality:
As you'll no doubt have read, ElfHosted is not a traditional seedbox, VPS, or shared hosting provider, but rather something new - we run the apps you choose, on a selection of appropriate hosts, in a highly-available and fault-tolerant way, using Kubernetes. And we open-source it all.
This distinction has many advantages (automation, scalability, resilience...), and some complex implications - for one, your apps don't necessarily run on the same hardware, so it may be that Radarr, Sonarr, and qBittorrent are all running on different hosts with different storage and processor capabilities. Our streamers (Plex, Jellyfin, Emby), for example, run on hosts with Intel Quicksync support, for hardware transcoding support.
We can't (easily) provide TCP-based access for services like SSH or FTP, which is why we rely on rlcone and WebDAV for data migrations, and we can't provide "root access" because each app runs on an as-locked-down-as-possible container, dispersed across our cluster.
One previously negative implication has been (until now...) that it's not simple to see app's real-time CPU / memory usage, or to watch logs in realtime. Certainly, these are available at a cluster-admin level, but without complicated authentication integrations, it's not possible to provide a user with visibility into their own resources, without exposing the privacy of other users.
We now have a solution I'm particularly proud of, because it exposes our Kubernetes environment on a per-user basis, in a way that's safe, while still providing all the cool visibility.. which I'm happy to now be able to share with you...
Create a custom CNAME on your domain name (noobflix.iamacouchpotato.com, for example), pointed to the FQDN of your elfhosted service (i.e. noobflix-overseerr.elfhosted.com), and the pick up a custom domain addon for Overseerr or Jellyseerr.
Plug your CNAME in, push the button, and within a few minutes we'll have generated the appropriate SSL certificate for your fancy new https://noobflix.iamacouchpotato.com instance!
Read on for details re changing the default logos too!
I made a post on LowEndSpirit yesterday, mentioning the 50% off on storage. This got some attention, resulting in an influx of new users, bumping us from 90 to 128 subscribers overnight!
The extra user load brought some issues to light, which I've spent today chasing down..
In a previous blog post, we discussed how we've expanded our node types to provide compute (CPU, RAM), storage, and network bandwidth for high-traffic-volume apps (torrent / NZB clients, data sync tools, etc) seperately from the resources for general-purpose and streaming apps.
In November, we provisioned 2 "download-class" nodes, but a month later, I've added a third node. This gives us an extra 1Gbps aggregate bandwidth capacity, provides better failover for workloads when doing maintenance, and allows for better automated balancing of workloads across the download-class nodes, since we can calculate a "mean" CPU/RAM/pod usage, and move workloads around accordingly.
Our platform is built on Hetzner auction server nodes, each of which has unlimited bandwidth, but is physically constrained to a 1Gbps NIC. As we've been bringing more subscribers on board, we've started to see occasional poor performance from the streamers (Plex, Jellyfin, Emby), most likely due to congestion on these 1Gbps links.
When you look at the graph below, the problem is immediately apparent (note the change when our maintenance window started):
So, today I added 5 more nodes to the cluster, to address two areas I felt were likely to become resource bottlenecks..
It turned out that due to changes in Radarr v4 compared to v5, our image wasn't passing testing, and so we didn't get the juicy new v5.0.3.8127 release a few weeks ago.
I was under the impression that v5 would support multiple resolutions for content, meaning a separate Radarr4K wouldn't be required anymore, but in the brief testing I've done so far, I've not been able to work out how to do this!
A longstanding issue we've been dealing with is that the Wordpress plugin we use to provide ElfBuckz doesn't include the ability to notify users when their balance is running low. So unless you manually keep track of your ElfBuckz balance, you're at risk of having your services stop when your ElfBuckz run out, without notice!
Since we recently fixed the deprovisioning bug, this has become even more important, so today I finally got my hands dirty and modified the plugin myself, adding the ability to notify users when their account balance goes low.
So.. you should now be alerted whenever your ElfBuckz balance drops below $1, which should give you plenty of time to arrange a topup (of course, once you subscribe to a top-up with the appropriate quantity, your ElfBuckz will "refill" every month, and you may never see these notifications)