"Elf-Disclosure" for May 2024
We're 10-months 
old now (or 1 year 
, if you don't count pre-production development time!), and May was spent on iterative growth and development, breaking new ground with video tutorials, and applying primarily behind-the-scenes and quality-of-life fixes.
To get us started, here are some shiny stats for May 2024, followed by a summary of some of the user-facing changes announced this month in the blog...
Stats
| Money | Mar 2024 | Apr 2024 | May 2024 |
|---|---|---|---|
| Cluster4 | $2540 | $1802 | $2925 |
| Store | $56 | $56 | $56 |
| CI | $100 | $100 | $100 |
| Cloud | $20 | $20 | $20 |
| Development 3 | 90h / $13,500 | 90h / $13,500 | 90h / $13,500 |
 OSS Sponsorship 5 | $35 | $35 | $35 |
| Total Expenses | $16,251 | $15,513 | $16,636 |
| Income | $1,525 | $2,021 | $1,630 |
| Income % of cash expenses | 53.14% | 100.4% | 52% 4 |
| Income % of all expenses | 9.38% | 13.02% | 9.8% |
| Focus | Mar 2024 | Apr 2024 | May 2024 |
|---|---|---|---|
| Subscribers | 413 | 360 | 338 |
| Ingress | 150TB | 256TB | 302TB |
| Egress | 58TB | 66TB | 59TB |
| Pods | 3446 6 | 3910 | 3188 |
| Focus | Mar 2024 | Apr 2024 | May 2024 |
|---|---|---|---|
| Unique visitors2 | 15.7K | 18.7K | 18.9K |
| Total pageviews2 | 54.9K | 58.5K | 56.3K |
| Discord members | 702 | 826 | 921 |
| YouTube subscribers | - | - | 72 |
| Focus | Mar 2024 | Apr 2024 | May 2024 |
|---|---|---|---|
| Total invested thus far 1 | $124,170 | $139,683 | $156,319 |
| Total revenue | $4,820 | $6,841 | $8,471 |
| Income % of total invested | 3.88% | 4.88% | 5.42% |
Resources
The stats below indicate CPU cores used (not percentage), and illustrate that after tenant workloads, our highest CPU consumer is traefik, which significantly spikes up during the daily maintenance window when there's a lot of pod "churn". We run Traefik as a daemonset, meaning we have > 22 pods, each of which gets "excited" when there's a lot of pod churn, and clocks in some heavy CPU usage, but this is still unexpected behaviour. Future work is planned to upgrade Traefik to v3 (it's a major update, needs careful testing), after which we'll revisit the CPU usage.
The "last" values on the chart are specific to when the snapshot was taken, but compared to the previous month, there's not a lot of change in overall tenant CPU usage (which is good, most of the resource pressure is on network and storage I/O). Observant readers will note that April's kubectl top nodes output includes 26 "elves", whereas May's output includes only 16, while the graphed CPU usage remains fairly consistent. This is part of an ongoing "right-sizing" effort!
kubectl top nodes
NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
dwarf01 266m 3% 23369Mi 73%
dwarf02 177m 2% 23381Mi 73%
dwarf03 167m 2% 23191Mi 72%
dwarf04 173m 2% 25477Mi 80%
dwarf05 176m 2% 23728Mi 74%
dwarf06 159m 1% 23654Mi 74%
dwarf07 239m 2% 23515Mi 73%
dwarf08 224m 2% 23253Mi 73%
dwarf09 273m 3% 23955Mi 75%
dwarf10 180m 2% 23216Mi 72%
elf01 869m 5% 31220Mi 24%
elf02 1113m 6% 32581Mi 25%
elf03 1224m 7% 36480Mi 28%
elf04 1365m 8% 45817Mi 35%
elf05 717m 4% 37240Mi 28%
elf06 1129m 7% 23484Mi 18%
elf07 2480m 15% 42267Mi 32%
elf08 1875m 11% 38316Mi 29%
elf09 978m 6% 44215Mi 34%
elf10 1212m 7% 34618Mi 26%
elf11 1663m 10% 38939Mi 30%
elf12 2959m 18% 36174Mi 28%
elf13 1734m 10% 42492Mi 33%
elf14 1489m 9% 34886Mi 27%
elf15 1728m 10% 34203Mi 26%
elf16 4718m 29% 51986Mi 40%
fairy01 2906m 18% 74833Mi 58%
fairy02 964m 6% 45500Mi 35%
fairy03 576m 3% 32683Mi 25%
giant01 1540m 12% 30243Mi 47%
giant02 862m 7% 27471Mi 42%
giant03 2462m 20% 31634Mi 49%
gnome01 1098m 13% 26761Mi 41%
gnome02 382m 4% 12228Mi 19%
gnome03 554m 6% 10844Mi 16%
goblin04 498m 4% 59342Mi 46%
goblin05 377m 3% 49317Mi 38%
goblin06 871m 7% 70292Mi 54%
Last month (Mar Apr)'s for comparison:
kubectl top nodes
NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
dwarf01 456m 5% 25932Mi 81%
dwarf02 316m 3% 24947Mi 78%
dwarf03 327m 4% 26507Mi 83%
dwarf04 353m 4% 26359Mi 82%
dwarf05 279m 3% 25536Mi 80%
dwarf06 323m 4% 26850Mi 84%
dwarf07 376m 4% 26702Mi 83%
dwarf08 290m 3% 25867Mi 81%
dwarf09 372m 4% 26528Mi 83%
dwarf10 328m 4% 26228Mi 82%
elf01 1533m 9% 28765Mi 22%
elf02 1958m 12% 33612Mi 26%
elf03 5858m 36% 26152Mi 20%
elf04 4438m 27% 59242Mi 46%
elf05 962m 6% 40678Mi 31%
elf06 996m 6% 30989Mi 24%
elf07 1856m 11% 36243Mi 28%
elf08 563m 3% 22437Mi 17%
elf09 1099m 6% 35952Mi 27%
elf10 1082m 6% 27806Mi 21%
elf11 2020m 12% 42999Mi 33%
elf12 1571m 9% 44910Mi 34%
elf13 1534m 9% 56472Mi 43%
elf14 2364m 14% 39203Mi 30%
elf15 3284m 20% 54181Mi 42%
elf16 915m 5% 35673Mi 27%
elf17 951m 5% 29772Mi 23%
elf18 2454m 15% 25660Mi 19%
elf19 775m 4% 45392Mi 35%
elf20 3122m 19% 31907Mi 24%
elf21 2299m 14% 41081Mi 31%
elf22 969m 6% 34268Mi 26%
elf23 1081m 6% 43334Mi 33%
elf24 1611m 10% 25941Mi 20%
fairy01 1618m 10% 81035Mi 62%
fairy02 902m 5% 49751Mi 38%
fairy03 969m 6% 35624Mi 27%
giant01 1249m 10% 25236Mi 39%
giant02 1096m 9% 25917Mi 40%
giant03 2496m 20% 25542Mi 39%
gnome01 373m 4% 13667Mi 21%
gnome02 623m 7% 27002Mi 42%
gnome03 1298m 16% 12507Mi 19%
gnome04 460m 5% 11687Mi 18%
goblin04 660m 5% 61026Mi 47%
goblin05 539m 4% 51855Mi 40%
goblin06 904m 7% 62636Mi 48%
This graph represents memory usage across the entire cluster. By far the largest consumers of RAM is rook-ceph, since ceph will basically take all the RAM you give it, for caching / performance.
RAM usage for tenants has again increased by ~25%, while CPU usage has remained relatively constant. This again indicate that there are more tenant apps in total (we know there are) which are idling most of the time, consuming RAM but not active CPU resources.
kubectl top nodes
NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
dwarf01 266m 3% 23369Mi 73%
dwarf02 177m 2% 23381Mi 73%
dwarf03 167m 2% 23191Mi 72%
dwarf04 173m 2% 25477Mi 80%
dwarf05 176m 2% 23728Mi 74%
dwarf06 159m 1% 23654Mi 74%
dwarf07 239m 2% 23515Mi 73%
dwarf08 224m 2% 23253Mi 73%
dwarf09 273m 3% 23955Mi 75%
dwarf10 180m 2% 23216Mi 72%
elf01 869m 5% 31220Mi 24%
elf02 1113m 6% 32581Mi 25%
elf03 1224m 7% 36480Mi 28%
elf04 1365m 8% 45817Mi 35%
elf05 717m 4% 37240Mi 28%
elf06 1129m 7% 23484Mi 18%
elf07 2480m 15% 42267Mi 32%
elf08 1875m 11% 38316Mi 29%
elf09 978m 6% 44215Mi 34%
elf10 1212m 7% 34618Mi 26%
elf11 1663m 10% 38939Mi 30%
elf12 2959m 18% 36174Mi 28%
elf13 1734m 10% 42492Mi 33%
elf14 1489m 9% 34886Mi 27%
elf15 1728m 10% 34203Mi 26%
elf16 4718m 29% 51986Mi 40%
fairy01 2906m 18% 74833Mi 58%
fairy02 964m 6% 45500Mi 35%
fairy03 576m 3% 32683Mi 25%
giant01 1540m 12% 30243Mi 47%
giant02 862m 7% 27471Mi 42%
giant03 2462m 20% 31634Mi 49%
gnome01 1098m 13% 26761Mi 41%
gnome02 382m 4% 12228Mi 19%
gnome03 554m 6% 10844Mi 16%
goblin04 498m 4% 59342Mi 46%
goblin05 377m 3% 49317Mi 38%
goblin06 871m 7% 70292Mi 54%
Last month (Mar 2024)'s for comparison:
kubectl top nodes
NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
dwarf01 456m 5% 25932Mi 81%
dwarf02 316m 3% 24947Mi 78%
dwarf03 327m 4% 26507Mi 83%
dwarf04 353m 4% 26359Mi 82%
dwarf05 279m 3% 25536Mi 80%
dwarf06 323m 4% 26850Mi 84%
dwarf07 376m 4% 26702Mi 83%
dwarf08 290m 3% 25867Mi 81%
dwarf09 372m 4% 26528Mi 83%
dwarf10 328m 4% 26228Mi 82%
elf01 1533m 9% 28765Mi 22%
elf02 1958m 12% 33612Mi 26%
elf03 5858m 36% 26152Mi 20%
elf04 4438m 27% 59242Mi 46%
elf05 962m 6% 40678Mi 31%
elf06 996m 6% 30989Mi 24%
elf07 1856m 11% 36243Mi 28%
elf08 563m 3% 22437Mi 17%
elf09 1099m 6% 35952Mi 27%
elf10 1082m 6% 27806Mi 21%
elf11 2020m 12% 42999Mi 33%
elf12 1571m 9% 44910Mi 34%
elf13 1534m 9% 56472Mi 43%
elf14 2364m 14% 39203Mi 30%
elf15 3284m 20% 54181Mi 42%
elf16 915m 5% 35673Mi 27%
elf17 951m 5% 29772Mi 23%
elf18 2454m 15% 25660Mi 19%
elf19 775m 4% 45392Mi 35%
elf20 3122m 19% 31907Mi 24%
elf21 2299m 14% 41081Mi 31%
elf22 969m 6% 34268Mi 26%
elf23 1081m 6% 43334Mi 33%
elf24 1611m 10% 25941Mi 20%
fairy01 1618m 10% 81035Mi 62%
fairy02 902m 5% 49751Mi 38%
fairy03 969m 6% 35624Mi 27%
giant01 1249m 10% 25236Mi 39%
giant02 1096m 9% 25917Mi 40%
giant03 2496m 20% 25542Mi 39%
gnome01 373m 4% 13667Mi 21%
gnome02 623m 7% 27002Mi 42%
gnome03 1298m 16% 12507Mi 19%
gnome04 460m 5% 11687Mi 18%
goblin04 660m 5% 61026Mi 47%
goblin05 539m 4% 51855Mi 40%
goblin06 904m 7% 62636Mi 48%
The 2 network graphs below to represent the two points where throughput varies.. the 10Gbps "giant" nodes (which are primarily used for ingress, as incoming Debrid content is passed to elves for streaming), and the 1Gbps "elf" nodes which are our general workhorses.
Compared to April's graphs, May's graphs appear to show slightly less traffic on the Elves - this may be due to optimizations preventing transcoding / unnecessary analysis in the streamers, or it may simply indicate usage patterns which vary from day to day (more streaming occurs on weekends).
Last month (Apr 2024)'s for comparison:
These are the traffic stats for egress from Hetzner. They exclude any traffic to/from Hetzner Storageboxes or ElfStorage, since this traffic is not classified as "external".
Ingress / Egress stats for May 2024
Last month (Apr 2024)'s for comparison:
Ingress / Egress stats for Apr 2024
Ceph provides our own storage ("[ElfStorage][elfstorage]"), typically used for long-term slow storage and seeding, as well as all the storage for our per-app /config volumes, now backed by 10Gbps nodes with NVMe disks.
Ceph storage usage grew ~15% during May, a three-fold increase over the previous month's growth rate.
Last month (Apr 2024)'s for comparison:
Retrospective
I iz YouTuber
In an effort to expand our user-base and reduce support overhead, I've started a YouTube channel explaining our various setups. I was unpleasantly surprised by how long it takes to post-edit a video, so I've settled on a workflow which is basically a one-take screencast, avoiding all the fancy edits tricks the "pros" do, and at all costs avoid the tired ol' click, subscribe, ring the bell catchphrase!
The most comprehensive of the videos recorded so far is this 24-min saga explaining how to configure plex_debrid:
After one of my videos was removed for violating community guidelines (I mentioned torrentio), I discoverd PeerTube, and created a mirror of the channel at https://video.elfhosted.com. (Also, hosted PeerTube instances may be an interesting product, let me know if it scratches your particular itch!)
While the video content hasn't necessarily boosted new signups significantly, I'd like to think that it's reduced the support overhead for new users! I have more videos planned for simple ("how to change your dashboard background") and complex ("how to use the aars") during June.
Charts and containers re-homed
Our helm charts (including the all-important "myprecious" umbrella chart) and container images are now all under the ElfHosted GitHub organization. This allows for more clearly delineated change management, isolation of privileges for automation, and nicer branding on social previews / links!
Improved versioning
Ever wonder what changed during the daily maintenance period? That chart change above means that now releases are versioned and tracked here. Soon (after some dust clears) these will be broadcast daily before the maintenance window, and you can always confirm your current version in the footer of your ElfHosted dashboard!
Support tickets opened up
While our #elf-help support tickets have won us global acclaim, one downside which has become apparent over time is that the solutions to common problems are "siloed" in individual support tickets, creating a burden / burnout effect on our brave ElfVengers.
To reduce the overhead on ElfVengers, and to surface common solutions to common problem, we've moved general support issues to Discord topics in #elf-support, which is soon to be improved with some clever "bottiness" inspired by the Kometa discord.
Individual account-related issues ("my subscription expired") are still handled in private tickets, but generally you'll get quicker assistance by referring to the #elf-support channel (also, previous topics are searchable, so you may find your answer!).
Rate-your-elf 
To assist with the process of gathering user reviews / testimonials, we've added a bot to capture user feedback. You'll notice these reviews surfacing at the bottom of our docs pages (look at the end of this report, for example). To add your review, just type /review into any Discord channel, and a friendly bot will prompt you.
Your feedback helps us to reach more Elves, so please keep it coming! 
Elf-errals earn you $5
After some positive feedback from our most prolific referrers (who can now easily cover their ongoing subs with the kickback from their referrals), we've updated and simplified our referral program as follows:
For every referral you make which turns into an order, you get a coupon for $5 off a purchase. 
If you have ongoing monthly subs, you can use this to stack ElfBuckz topups to fund those subs.
To get your own unique referral link, visit https://store.elfhosted.com/my-account/myreferrals
Given the majority of our users are already familiar with our tools, an effective way to get visibility on your referral link is to respond to questions on social media, like in adjacent subreddits like r/realdebrid, r/seedboxes, r/plex, etc, clearly disclosing the nature of the referral link :)
TRaSH is TReaSUre
The promised details on using TRaSH guides with your ElfHosted apps are now published. You can do it the easy way (elfbot recyclarr sync) or the hard (self-inflicted) way!
AirDC++ "inflates"
AirDC++ was announced, providing yet another fancy UI to keep your inner geek entertained!
What's coming up?
Riven rises
Beloved Iceberg dev @spoked has fallen for our beguiling ElfHosted platform 
, and will be building the "next" version of Iceberg ("Riven") with ElfHosted as a first-class, tier-one ecosystem!
(Iceberg is a functional "reboot" of plex_debrid with a beautiful web UI)
This means ElfHosted users will realize "maximum gainz" of Riven immediately, and optionally be able to replace the "Advanced Infinite Streaming" Arr stacks (Radarr+Sonarr+Prowlarr+Scanarr+RDTClient) with a single app!
PlexTraktSync
PlexTraktSync is a tool which syncronises a Plex library with a Trackt library. Why would you want such a thing? One option we're exploring is to transition a user from "Infinite Streaming" (plex_debrid-based) to "Advanced Infinite Streaming" (aar-based), which would require re-creating Plex libraries. The idea would be to use Plextrackt to sync a Trakt list with an existing Plex library, and then feed that Trackt list to Radarr/Sonarr, to bootstrap the arr-based library collection.
Storage re-refactoring
In Mar 2024, we migrated some of the high-I/O-consuming apps off of our CephFS storage layer, to standalone Ceph RBD block devices, to avoid metadata bottlenecks. While this massively improved performance for Plex and the Aaars, the downside is that it's no longer possible to manage the apps' config via FileBrowser, which creates additional support overhead when (for example) the Aars go rogue and decide to perform a scheduled backup once a minute, filling up /config and crashing the pod!
During June, we'll work on a way to regain storage visibility across all the apps, without sacrificing performance to the gods of CephFS!
PostgreSQL support for the Aars
PostgreSQL aar support on ice 
Update: While PostgreSQL support for the Aars sounds promising, the associated user support overheads are not insignificant (how will a user backup/restore, for example?), and with the imminent arrival of Riven, the Aars may loose their prominence in our stack. So PostgreSQL on the aars is "on ice" for now. I'll remove this section from future reports..
Radarr, Sonarr, etc all now have built-in support for PostgreSQL as a replacement to the troublesome and easy-to-corrupt SQLite database they come with by default. To support our KnightCrawler database, I've started using CloudNativePG (CNPG) for full-lifecycle database management. With a simple CR, CNPG will establish a highly-available PostgreSQL cluster, including automated failover, incremental and automatic backups to local snapshots and to an S3 bucket.
This declarative approach to creating a PostgreSQL database could allow us to, in bulk, create individual database for Arr instances, such that setup is still fully automatic, and users just "get" the PostgreSQL-enabled instances. Once again deferred by more pressing issues in April, I hope to pay more attention to this in ...\
Offering free demos (
AudioBookshelf)
Audiobookself demo coming soon
I've offered a free demo instance to the AudioBookshelf community, in line with the plan below. We should see this eventuate during June!
Nothing gets my attention on a new app like a live demo. I've considered reaching out to open source projects who don't have their own online demo, and offering to host a self-resetting demo for them.
This approach has been successful with the Stremio Addons, and I think it'd be effective at gaining exposure to more niche communities.
A hosted demo would provide their potential users the opportunity to evaluate the app "live", and would also drive more traffic / recognition / SEO juice towards ElfHosted.
If you've got an open-source, self-hosted app and you'd like a free demo instance hosted, hit me up!
(We are currently donating torrent hosting to https://freerainbowtables.com)
Your ideas?
Got ideas for improvements? I'd love to hear them, post them in #elf-suggestions!
How to help
If you'd like to make a donation in recognition of our infrastructure costs, our open-source resources, or our friendly support, a simple donation product is available at https://store.elfhosted.com/product/elf-love/ 
Another effective way to help is to drive traffic / organic discovery, by mentioning ElfHosted in appropriate forums such as Reddit's r/selfhosted, r/seedboxes, r/realdebrid, and r/StremioAddons, which is where much of our target audience is to be found!
Join us!
Want to get involved?
Want to get involved? Join us in Discord and come and test-in-production!
What users say..
Here's what some of our usersfriends say..
"The Elfhosted product alone is great. The team is super transparent on the issues and resolutions that happen as it grows. Lots of opportunity to self-solve issues, but the support from both mods and darth-penguini is SO quick and proactive. Props to them."
"I don't understand how there aren't tons of elves already. The way this whole thing is setup is great, from getting a free $10 amount of elfbuckz to start, free 7 day trials on most services, the subscription prices of the products are reasonable, to the smoothness of how everything runs. I just dont get it, people choose to pay way more with places that don't provide the same customer service nor have a great community like this one. I recommend elfhosted to everyone.
Pricing update since this review
Our pricing model was updated in July 2024, removing the free $10 ElfBuckz and the 7-day trials, and replacing them with standard pricing tiers and a 14-day refund policy.
"Just wanted to check in here and let @Darth-Penguini and anyone/everyone else know...WOW. I have been struggling with storage for years, maintenance of Docker containers, upkeep, all of it. Elfhosted is so freeing. It's an amazing service that I hope to be a member of for a long, long time!"
"I just have to say, I am an incredibly satisfied customer. I had been collecting my own content for nearly 20 years. Starting off with just a simple external HD before eventually graduating to a seedbox with 100TB of cloud storage attached and fully automated processes with Sonarr and Radarr . However, the time came when the glory days of unlimited Google Drive storage ended. I thought my days of having my full collection at my fingertips via :plex: were behind me, until I found Real-Debrid and ElfHosted.
Now I essentially have the exact same access to content as I had before, but even better. Superior support and community involvement. Content is available almost immediately after being identified. A plethora of tools at my fingertips that give me more control and automation than ever before. Wonderfully well done and impressive! I am looking forward to being a customer for a very long time! Massive kudos to @funkypenguin 🤟
"I would recommend ElfHosted to anyone. It has been great so far and made life a lot easier than running my own setups. If you’re in the fence give them a try and help support this great community."
"thanks for the help and must say this is the best host I every had for my server 🙂 10/10 🙂 All other places I have try have I got a lot buff etc. Your host can even give me full power on a 4K Remux on 200GB big movie file . That's damn awesome 😄"
@tjelite (Discord)
"What an amazing support system these guys have Chris and Layzee i think it was! Both are very patient with me even though I am a newbie at all this. Very thorough and explained everything step by step with me
I couldn’t ask for anything better than the service I have received by these guys! Happy happy client❤️"
"Very happy customer. Great service"
"Very good customer service, frequent updates, and excelent uptime!!!!!"
"I had my own plex-arrs setup on hetzner for years. Yesterday I deleted everything as elfhosted has gone above and beyond it. And it has a fantastic, active community as well! Very friendly, helpful and like-minded folks always willing to help and improve the system. Top notch!"
"Absolutely Amazed with the patience and professionalism of all Elf-Venger Staff including bossman penguin❤️"
"@BSM went above and beyond to make sure I had all the one on one support needed with my sub. Thank you for your patience! Elfhosted continues to be Elftastic !!"
"really enjoying the service from elfhosted. The setup is really easy from the guides on the website. And the help on the discord channel is really quick."
"Support is amazing, and once you find a setup which works best for you it works perfectly, very happy 😊"
"great fast service, resolved my problem and really friendly"
"Great service and sorted out a billing issue super quick and easy."
"Very friendly support, resolved a problem with my account! I also appreciate the community that has been built around the service!"
"excellent customer service and very fast replies"
"Loved the simplicity, experience and support"
"Very friendly help as always, problem solved, one happy elf here!"
"Great Helpful and Fast support. Thanks!"
-
All moneyz are in US dollarz ↩
-
For simplicity's sake, we're only presenting stats for https://elfhosted.com here, and not https://store.elfhosted.com. Fully transparent traffic details are available for elfhosted.com and store.elfhosted.com ↩↩
-
Includes "opportunity cost" of deferring billable consulting work for ElfHosted development! ↩
-
The month-on-month variation here may be explained by the change in Hetzner billing models, the fallout of which was only completed in May ↩↩
-
These are actually paid yearly in advance, but represented here monthly for consistency. Confirm my sponsorships here ↩
-
Total pod count dropped because we (a) improved pruning of un-subscribed services, and (b) removed rclonefm/ui by default, but made them free to add, in the store. ↩