"Elf-Disclosure" for Dec 2023
We made it through the holiday season! With 5 months in production now, and continued growth, December's focus was refactoring our storage design to accommodate said growth, which resulted in some heavy hardware investments, as you'll note below.
Black Friday and end-of-year storage sales also contributed to increased subscriber growth towards the end of the year, and an increase in effort spent on stability and scalability.
This month we gained an AI support bot, a handful of new apps, and performance / stability improvements especially for the downloaders / streamers..
To get started, here are some shiny stats for Dec 2023, followed by a summary of some of the user-facing changes announced this month in the blog...
Stats
| Focus | Oct 2023 | Nov 2023 | Dec 2023 |
|---|---|---|---|
| Cluster | $428 | $428 | $1330 |
| Store | $56 | $56 | $56 |
| CI | $100 | $100 | $100 |
| Cloud | $20 | $20 | $20 |
| Development 3 | 15h / $2,250 | 30h / $5,500 | 45h / $6,750 |
| Total Expenses | $2,854 | $6,104 | $8,256 |
| Income | $318 | $229 | $505 |
| % of expenses | 11.1% | 3.75% | 6.1% |
| Focus | Oct 2023 | Nov 2023 | Dec 2023 |
|---|---|---|---|
| Subscribers | 48 | 86 | 124 |
| Ingress | 34TB | 20TB | 22TB |
| Egress | 14TB | 120TB | 70TB |
| Pods | 516 | 794 | 1002 |
| Focus | Oct 2023 | Nov 2023 | Dec 2023 |
|---|---|---|---|
| Unique visitors2 | 2.4K | 4.9K | 5K |
| Total pageviews2 | 6.8K | 15.5K | 18.6K |
| Discord members | 128 | 162 | 199 |
| Focus | Oct 2023 | Nov 2023 | Dec 2023 |
|---|---|---|---|
| Total invested thus far 1 | $58,965 | $65,069 | $71,819 |
| Total revenue | $671 | $900 | $1,405 |
| Income % of total invested | 1.13% | 1.38% | 1.96% |
Resources
Most apps consume almost no CPU while idle - the larger consumers are streamers / Tdarr doing transcoding, download clients doing download/unpack operations.
CPU usage effectively doubled from last month as graphed. While we did add additional storage nodes (and therefore OSD pods) during the period, these nodes are heavy on RAM and light on CPU, so the CPU growth we're seeing is most likely the actual impact of increased subscribers / trial users, and the fact that download clients now have more "room to run".
Last month (Nov 2023)'s for comparison:
This graph represents memory usage across the entire cluster. By far the largest consumers of RAM is rook-ceph, and all the extra storage nodes which were added would explain the biggest increase in RAM, since ceph will basically take all the RAM you give it, for caching / performance.
Ignoring ceph though, RAM usage has increased from ~75GB to ~90GB, a ~20% increase, which is less than the 50% CPU increase we noted. Once we stop adding hardware, we'll get a decent memory usage baseline for growth!
Last month (Nov 2023)'s for comparison:
I'm not sure these stats are accurate, they've likely overly high because pods on the host network (like metallb, ceph, etc) will end up counting all traffic on each host, rather than the pod itself. This is exacerbated with more and more storage nodes, which run in `hostNetwork`` mode. However, the graph still gives a good indication of network usage compared to the previous month.
Like last month, the first indicator of increased network usage is that the vertical scale on our graphs has now more than doubled :) Ingress / egress has not significantly changed this month, which probably indicates that increased network usage is a result of improved network storage capacity.
Last month (Nov 2023)'s for comparison:
These are the traffic stats for egress from Hetzner. They exclude any traffic to/from Hetzner Storageboxes.
Here we see a predictable pattern of ingress over egress, and evidence via the traffic counters, that the great majority of our network traffic is as a result of the downloaders on orcs01-3. We wouldn't see the resulting network traffic to ElfStorage or Storageboxes though, since this traffic is not counted as being "external".
One point to note is how under-utilized the dwarves (storage nodes) are, WRT internet traffic. In Jan 2024, we've started experimenting with moving high-throughput, low-resource applications (i.e. RDTClient) onto these nodes to take advantage of the unused capacity.
Last month (Nov 2023)'s for comparison:
Ceph provides our own storage ("ElfStorage"), typically used for long-term slow storage and seeding.
Nov 2023's report noted:
At this rate, we'll run out of storage in a few months. There's some outstanding dev work required re scrubbing of old, abandoned volumes, which may be inflating our usage stats, but once we've eliminated any abandoned data, it may be time to plan for disk expansion within the next months.
Storage consumption was the biggest change in December 2023. Once the rate of growth became clear, we expanded to 10 storage nodes, and created a new erasure coded pool for new users (pre-existing users remain on the less-efficient 2-replica pool).
The gauges below can be misleading - in Nov, the 28.9TB used represents ~15TB of actual use, since the pool would simply keep 2 copies of every object for redundancy (i.e, 100% overhead). The transition to erasure coding means only a 12.5% overhead (we split every object into 8 chunks, distribute them to 8 separate nodes, and keep 2 "coding" on a further 2 nodes for redundancy). As a result, the Dec 2023's usage of 58TB is more than an actual growth from 15TB to 25TB - it's likely at least a 100% increase, since significant pruning of old volumes occurred in the Dec period.
Last month (Nov 2023)'s for comparison:
What's new?
Rebranding!
We have a new mascot, with some updated branding, which now graces your dashboards, our docs, store, and emails.
(I asked for "nerdy elf holding servers")
All hail Glime, our new AI overlord
After a brief trial period, I'm now officially an early customer of Glime, an integration which parses our docs / Discord history, and feeds them to an AI (ChatGPT currently) to answer questions in a chat-based, human-sounding context.
You can ask Glime anything about ElfHosted either via the widget on the website, or in Discord, by typing "@Glime how do I
The big advantage of Glime, from my perspective, is that it does the hard work of parsing our docs, so that you can (without waiting on my NZ timezone) simply ask questions like "how do I connect Radarr to NZBGet?", and receive an ElfHosted-specific answer!
Stream from Real-Debrid with Zurg
Zurg is "middleware", which lets you mount a Real-Debrid account into /storage/realdebrid-zurg.
Why would you want to do this?
Using Zurg means you can stream content directly from Real-Debrid, without having to download it first to attached storage (usually a job for RDTClient), by just adding /storage/realdebrid-zurg to your streamer's libraries.
It's not exactly battle-tested yet, being so new, but I've successfully streamed a portion of a 4K movie via Plex, without having to worry about where to store it!
Instant library updates with autoscan
All the streamers now have Autoscan built in, so your Arrs can now trigger a targetted, partial scan, whenever new media becomes available.
See this blog post for details!
Data retention policy documented
For clarity's sake, our auto-deletion / data retention policy is now clearly documented.
TL;DR : We retain your config for 7 days, and your data for 31 days, unless you explicitly trigger an account deletion, in which case we nuke them immediately.
1TB ephemeral /tmp on downloaders
Since our downloaders now run on dedicated nodes with heaps of local HDD, we provide 1TB of ephemeral storage in /tmp for temporary unpacking, repair, torrent-assembly, etc. This can significantly improve the speed of storage operations, while reducing unnecessary load on network storage.
See this blog post for details!
Wallabag and Homer added to the flock
In December, we added Wallabag (a read-it-later app), and Homepage (a beautiful, alternative app dashboard) to the app catalogue. See the individual app pages for details.
How to help
If you'd like to make a donation in recognition of our infrastructure costs, our open-source resources, or our friendly support, a simple donation product is available at https://store.elfhosted.com/product/elf-love/ 
Another effective way to help is to drive traffic / organic discovery, by mentioning ElfHosted in appropriate forums such as Reddit's r/selfhosted or r/seedboxes, which is where much of our historical our target audience is to be found!
What's coming up?
"Dummies" guides to app stacks
Despite having implemented RDTClient a few months ago at user request, I only recently discovered Real-Debrid, and worked out how it's could save me from all the hassle of torrenting, seeding, fretting about ratios, etc. It took a while to get my head around the process, so I'm working on a user guide (with a flowchart!) to make the process easier.
My suspicion is that more users would use more apps, if they only knew how to integrate them with their current workflows / requirements.
More #elf-help coverage
With the increase in users over the past month, it's becoming hard to field support issues by myself, even with Glime AI to help out. A few users have offered to help out with user support / onboarding, and I'm busy working out how this will work, and how ElfHosted can reciprocate.
If you'd like to help cover tier-1 / tier-2 user support issues, hit me up in a support channel (#help-ception!) with your ideas! :)
Surfacing of non-mainstream apps
Every time I review our apps list, I notice apps that are not getting much attention, since they're not the mainstream "bread and butter" media / downloading apps. Some very useful apps like Notifiarr, RPBD, OpenBooks, etc, aren't often purchased, and I wonder if it's because they're not well-known enough.
I've been thinking about how to "showcase" these apps, and have considered options like a detailed "app of the week" post, or an automate sequence of emails for new users, showcasing less well-known apps and ElfHosted features.
(Nothing happened with this during December, but it's still on the radar)
Offering free demos
As I confessed above, nothing gets my attention on a new app like a live demo. I've considered reaching out to open source projects who don't have their own demo, and offering to host a self-resetting demo for them.
This would provide their potential users the opportunity to evaluate the app "live", and would also drive more traffic / recognition / SEO juice towards ElfHosted.
If you've got an open-source, self-hosted app and you'd like a free demo instance hosted, hit me up!
(We are currently donating torrent hosting to https://freerainbowtables.com)
Your ideas?
Got ideas for improvements? I'd love to hear them, post them in #elf-suggestions!
Join us!
Want to get involved?
Want to get involved? Join us in Discord and come and test-in-production!
-
All moneyz are in US dollarz ↩
-
For simplicity's sake, we're only presenting stats for https://elfhosted.com here, and not https://store.elfhosted.com. Fully transparent traffic details are available for elfhosted.com and store.elfhosted.com ↩↩
-
Includes "opportunity cost" of deferring billable consulting work for ElfHosted development! ↩