"Elf-Disclosure" for November 2023
We've been in prod for 4 months now, and with positive organic subscriber growth, we ran into some scaling issues during November 2023, which were addressed by both further infrastructure investment and some architectural decisions (discussed in this blog post)
First off, here are some shiny stats for November 2023, followed by a summary of some of the user-facing changes announced this month in the blog...
Stats
Resources
Most apps consume almost no CPU while idle - the larger consumers are streamers doing transcoding, and download clients doing download/unpack operations.
CPU usage was fairly similar to last month as graphed, but given that we increased from 9 nodes to 14 nodes, the actual increase from last month is probably hidden, and is likely around 30%.
Last month (Oct 2023)'s for comparison:
This graph represents memory usage across the entire cluster. By far the largest consumers of RAM is rook-ceph, and since we added 6 more OSDs across 3 hosts this month, the increase in the biggest RAM consumer is expected.
Ignoring ceph though, RAM usage has increased from ~55GB to ~75GB, or a ~30% increase, relatively consistent with last month's growth.
Last month (Oct 2023)'s for comparison:
I'm not sure these stats are accurate, they've likely overly high because pods on the host network (like metallb, ceph, etc) will end up counting all traffic on each host, rather than the pod itself. However, the graph still gives a good indication of network usage compared to the previous month.
The first indicator of increased network usage is that the vertical scale on our graphs has doubled :) The network graphs indicate roughly a 40% increase in network usage, which roughly aligns with the ingress/egress traffic stats.
Last month (Oct 2023)'s for comparison:
These are the traffic stats for egress from Hetzner. They exclude any traffic to/from Hetzner Storageboxes.
The big traffic spike which started on 15 Oct ended around 15 Nov, but given it was half the month (roughly the same as last month), there's still an overall increase of 40% total throughput over the previous period.
Last month (Oct 2023)'s for comparison:
Ceph provides our own storage ("[ElfStorage][elfstorage]"), typically used for long-term slow storage and seeding. While the read/write IOPS and throughput gauges below are useless for trending (they're just a single point in time, I'm working on adding better metrics), the capacity graph indicates that we've consumed 75% more capacity compared to October 2023 (12TB vs 28TB).
At this rate, we'll run out of storage in a few months. There's some outstanding dev work required re scrubbing of old, abandoned volumes, which may be inflating our usage stats, but once we've eliminated any abandoned data, it may be time to plan for disk expansion within the next months.
The Ceph upgrades we undertook during Nov also changed our dashboard, to a far less-combined view. The following summarizes the current capacity / load on the cluster, and hopefully better stats will be available in Dec, after the introduction of Loki for long-term metric aggregation.
Last month (Sep 2023)'s for comparison:
What's new?
Investment in resource isolation
Our platform is built on Hetzner auction server nodes, each of which has unlimited bandwidth, but is physically constrained to a 1Gbps NIC. As we've been bringing more subscribers on board, we've started to see occasional poor performance from the streamers (Plex, Jellyfin, Emby), most likely due to congestion on these 1Gbps links.
When you look at the graph below, the problem is immediately apparent (note the change when our maintenance window started):
Recently we added 5 more nodes to the cluster, to address two areas I felt were likely to become resource bottlenecks..
More details in this blog post.
Pre-empting disk failures
Our nodes are running on "second-hand" dedicated servers, purchased on auction from Hetzner. We had a our second disk replacement support case in November, when one of the drives failed (and unfortunately, I initially identified the incorrect drive as the culprit!).
I decided to no longer find out about disk failure issues after the fact, and worked (during November) to add better S.M.A.R.T metrics to our observability stack. Check out the result - some of these drives are nearly 9 years old!
Needless to say, I'll be working on asking Hetzner to replace some of these drives, based on the metrics we now have available!
How to help
Today I was grateful when a new subscriber (after reviewing last month's Elf-Disclosure report) asked whether we had a donations page (we didn't, but we do now!), given our expenses still outweigh profit by 2500% 
Thanks to this generous subscriber, these numbers are tracking to look slightly better in next month's report! 
If you'd like to make a donation in recognition of our infrastructure costs, our open-source resources, or our friendly support, the page is available at https://store.elfhosted.com/product/elf-love/ 
Another effective way to help is to drive traffic / organic discovery, by mentioning ElfHosted in appropriate forums such as Reddit's r/selfhosted or r/seedboxes, which is where much of our historical our target audience is to be found!
What's coming up?
More options for encrypted mounts
Currently selling encrypted rclone mounts is a complex, manual process, because the rclone config file requires two remotes, and at least 3 obfuscated "secret" values. With some motivation from subscribers, I'm working on adding rclone obfuscation to our provisioning process, so that users can simply enter the necessary credentials when they purchase an encrypted mount (like we do with storageboxes), and have the mount automatically setup.
Surfacing of non-mainstream apps
Every time I review our apps list, I notice apps that are not getting much attention, since they're not the mainstream "bread and butter" media / downloading apps. Some very useful apps like Notifiarr, RPBD, OpenBooks, etc, aren't often purchased, and I wonder if it's because they're not well-known enough.
I've been thinking about how to "showcase" these apps, and have considered options like a detailed "app of the week" post, or an automate sequence of emails for new users, showcasing less well-known apps and ElfHosted features.
I thought I might do some sort of "Christmas countdown", introducing new apps on a daily basis leading up to Christmas. I'm a little wary of the time commitment though! :)
Sexier dashboard
I'm a sucker for shiny demos, and recently fell for "Homepage", a YAML-based (yay!) beautiful dashboard, with lots of app integrations (many more than our default, Homer).
I'll post a separate blog post about this in more detail, but if you'd like your dashboard to look a little more like this...
... then check out Homepage!
Offering free demos
As I confessed above, nothing gets my attention on a new app like a live demo. I've considered reaching out to open source projects who don't have their own demo, and offering to host a self-resetting demo for them.
This would provide their potential users the opportunity to evaluate the app "live", and would also drive more traffic / recognition / SEO juice towards ElfHosted.
If you've got an open-source, self-hosted app and you'd like a free demo instance hosted, hit me up!
(We are currently donating torrent hosting to https://freerainbowtables.com)
Your ideas?
Got ideas for improvements? I'd love to hear them, post them in #elf-suggestions!
Join us!
Want to get involved?
Want to get involved? Join us in Discord and come and test-in-production!