"Elf-Disclosure" for July 2023
Our little 
is growing up, having passed our awkward adolescent "alpha" stage (June 2023), and slightly less awkward "beta" stage (July 2023), we're "all grown up" and ready for customers, in Aug 2023 
Looking back at July, here were the major changes...
What's new?
Switched to daily subscriptions
Early testing showed that (a) users want to add/change apps frequently, until the get their workflow "dialled in", and (b) Wordpress + Woocommerce isn't great at managing this, since its subscription plugin is primarily built for physical subscriptions, where it's reasonable to expect that a change to your subscription would only take effect on the next renewal date.
To avoid tricky problems with prorating / refunds, and to further differentiate ourselves, we switched to daily subscriptions. There are still subscription edge cases where you end up with a new app for a fraction of a day for free, or you loose out the remainder of that app's availability if you cancel it before your subscription refreshes, but since it's costing a fraction of $0.05 ElfBuckz, I don't think anybody cares too much!
24h trial apps
An advantage afforded by daily subscriptions is that we can also offer "one-time" apps purchases. A one-time (24h) app will get you an app for at least 24h (sometimes a little more, depending on automated task timing), thereafter the app will be automatically removed.
Potential use cases for one-time apps are:
- Trial an app without having to worry about cancelling it from your subscription later
- Run an app only while you need it (i.e., archiving a YouTube channel with YouTubeDL)
Storage for 24h apps will be auto-removed after 7 days
Any config storage provisioned for a 24h app purchase will be auto-removed after 7 days. So it wouldn't be possibel, for example, to run Jellyfin on the 1st of every month for a day, unless you make alternative arrangements to backup/restore its config inbetween.
Added ElfBuckz
A problem with daily subscriptions (or even cheap monthly ones) was that there's a minimum amount that our payment processors (Stripe, PayPal, etc) would require in a transaction, and every transaction would be subject to fees.
An innovative solution ended up being "ElfBuckz", our in-store currency, which we can (a) give away on signup, and (b) top up with real money.
So the most efficient way to do our daily subscriptions is to pay with an ElfBuckz balance, and then to top up ElfBuckz as necessary, or on a regular monthly basis.
Refactored /config storage into per-app volumes
It's not a user-visible change, but we switched from providing a single, 50Gi /config volume to providing per-app /config/ volumes. This aligns better with our per-app resourcing (apps are resourced appropriately for reasonable use), and it lets us offer more granular storage options, as well as expiring storage for 24h apps (see warning above).
Removed LongHorn
The LongHorn components in the cluster weren't keeping up even when we had 1 x 50Gi /config volume per-tenant, and they definitely wouldn't have handled exploding this into potentially 30+ /config volumes per user as described above. Partly the reason for this is to provide RWX volumes (which we need), LongHorn runs a "share" pod for every PVC which needs to be accesesd by more than one pod simultaneously, running a little Ganesha NFS server. A clever solution for small-scale, but not a desgin which would scale as we exponentially increased users and volumes!
So what are we using instead?
We're using plain ol' Rook Ceph, the same cluster which provides our ElfStorage volumes. Ceph has the advantage of kernel support, so there's no need to add another pod per-PVC, and while it may not be super-fast/performant on our Hetzner hardware, it's reliable and horizontally scalable.
Removed ElfVPN
ElfVPN was a shared VPN provided by default to the torrent clients, for users who wanted to "test the waters" before going with BYOVPN. It was never intended to be performant or scalable, but to provide a good "first impression". Within a week of announcing ourselves to r/seedboxes, the VPS running ElfVPN was suspended for DMCA-related abuse, and we decided not to replace it.
Torrent clients now require the user to BYO VPN credentials - We've got pre-configured products for various popular VPN providers, and we can quickly add more upon request.
Switched CNI to Cilium
Another behind-the-scenes update - after careful testing on the CI cluster, we live-migrated our CNI from flannel+wireguard to Cilium. The primary motivaton was to gain access to Cilium's Egress Gateway feature.
Egress Gateway will hopefully give us the ability to configure which node our egress traffic leaves the cluster from (regardless of which node the workload is running on), to better balance our egress traffic limits from Hetzner (20TB/month/node before I have to pay extra!)
Open Sourced all the things!
True to our original mission, we open-sourced the infrastructure and helm chart repos which run the cluster and provide user services.
Stats
Here's our stats, updated for July 2023:
| Focus | June 2023 | July 2023 |
|---|---|---|
| Cluster | $428 | $428 |
| Store | $632 1 | $223 |
| CI | $208 | $200 |
| Cloud | $30 | $20 |
| Development | 146h / $21,900 | 124h / $18,600 2 |
| Focus | June 2023 | July 2023 |
|---|---|---|
| Users | 14 | 48 |
| Ingress | 24TB | 19.5TB |
| Egress | 1TB 4 | 3.3TB |
| Pods | 478 | 619 |
| Focus | June 2023 | June 2023 |
|---|---|---|
| Total invested thus far | $23,200 | $42,669 6 |
| Revenue | $0 | $43 (0.1% of total spend!) |
Resources
Note
We lost our monthly metrics when migrating from Longhorn to Rook-Ceph, so what's below indicates usage at the end of the period.
Most apps consume almost no CPU while idle - the larger consumers are streamers doing transcoding, and download clients doing download/unpack operations:
Last month (June 2023)'s for comparison:
This graph represents memory usage across the entire cluster. By far the largest consumers of RAM is rook-ceph:
Last month (June 2023)'s for comparison:
I'm not sure these stats are accurate, they've likely overly high because pods on the host network (like metallb, ceph, etc) will end up counting all traffic on each host, rather than the pod itself. This is an outstanding issue to fix!
Last month (June 2023)'s for comparison:
These are the traffic stats for egress from Hetzner. They exclude any traffic to/from Hetzner Storageboxes:
Last month (June 2023)'s for comparison:
Ceph provides optional storage ("ElfStorage"), typically used for long-term slow storage and seeding:
Join us!
Want to get involved?
Want to get involved? Join us in Discord and come and test-in-production!
-
Much of this is yearly fees for Wordpress plugins ↩
-
Yes, that's a lot! This is the opportunity cost, over a month, of focusing on ElfHosted rather than billable consulting work! ↩
-
Total spend includes yearly payments for Wordpress plugins, etc ↩
-
Low egress is good, because ingress is always free, but Hetzner charges for egress after 20TB! ↩
-
Except Minio, which we're not bringing back! ↩
-
All moneyz are in US dollarz! ↩