Clarifying Costs of Running the Fediverse with Jerry from Infosec.Exchange
![[avatar]](/pictures/favicon.png)
Blender Dumbass👁 591
https://pawb.social/ : 👁 1
https://lm.madiator.cloud/ : 👁 1
https://piefed.social/ : 👁 5
https://phanpy.social/ : 👁 4
https://blenderdumbass.org/ : 👁 6
https://lemmy.dbzer0.com/ : 👁 2
https://communick.news/ : 👁 1
https://community.nodebb.org/ : 👁 2
https://feddit.org/ : 👁 3
https://quokk.au/ : 👁 1
https://discuss.tchncs.de/ : 👁 1
https://phtn.app/ : 👁 2
https://lemmy.ca/ : 👁 2
https://mastodon.social/ : 👁 6
https://t.lemmy.dbzer0.com/ : 👁 1
https://blenderdumbass.org/articles : 👁 3
https://mastodon.online/ : 👁 2
https://lemmy.blahaj.zone/post/27498554?scrollToComments=true : 👁 1
https://duckduckgo.com/ : 👁 2
https://blenderdumbass.org/articles/help_me_collect_financial_statistics_of_the_libre_community : 👁 2
https://phanpy.outmo.de/ : 👁 1
https://blenderdumbass.org/search?text=%23freesoftware&tags=on : 👁 1
Includes an exclusive interview with Jerry.
My last article "We have to solve the money problem!" caused some chaos. It is one of the most read articles I've ever published. And for a good reason. I started it by claiming this:
On the Fireside Fedi interview with Jerry ( the admin of Infosec.Exchange Mastodon instance ) a scary truth was suddenly revealed ( on 34:11 ): Just to keep the instance up and running he needs to spend up to $5000 a month, pretty much out of his pocket. Donations to the instance barely cover any of that. And if he will ask people to pay to use it, they will, rightfully so, switch to a different instance.
I have posted the article to a few places on the internet. And we had wonderful discussions about various things we could do to solve that problem. If you want to see what people have to say about that problem look at:
- Fediverse at lemmy.word 100+ comments.
- Free and Open Source Software at beehaw.org
- Open Source at lemmy.ml
- BT Free chat on Matrix
Under the article itself ( since I started implementing Activity Pub directly into the blog software ) some people also commented.
One of those people was @jerry@infosec.exchange himself who said the following.
@blenderdumbass just to be clear, it was out of my pocket at one point, but the costs have been covered by donations for a long time now. Sorry if I didn’t make that clear on the show.
That is very good news!
It seems like the fediverse is not dying, but we still need to think about scalability. In the Lemmy post with 100+ comments Jerry had commented this:
I think the donation model is working ok at this scale, but I don’t believe it will scale up to the hypothetical future we were discussing on the show where the fediverse became the social media platform for the masses.
He goes on for a bit to illustrate his point mathematically.
My conversation with Jerry
I decided since I don't understand how all of this works, I will just simply ask Jerry personally about all of this data and technical details, so that people will no longer be confused about all of this.
Jerry let me correct typos in his answers. But other than that I used very minimal editing. The original answers will be linked too.
I did that on mastodon in a public thread.
I asked:
Are there any places where there is a detailed report people can look at, explaining all the various numbers related to the money problem of your instances? If there isn't: How much you get from donations? How much you pay for each of the instances? How many people are on each of the instances? And what exactly are you paying for? Is it hosting? It is processing power? What is it that is so expensive?
Jerry answered with this:
I don’t use separate servers for the various instances, so I don’t have a breakdown of costs by instance type. My Hetzner bill is about $3200/month, though it fluctuates wildly lately due to the roller coaster foreign exchange rates, however it’s been trending up lately. That cost covers 9 rented servers. The majority of the cost is in the 4 Dell servers: 3 of the DX182’s and one DX293, as well as one AX102 that has extra NVME storage to host a minio server all on a 10G network. Shortly after the mass migration from Twitter to Mastodon, I tried scaling horizontally, running on lower end Hetzner systems and it was a lot of work. So, I eventually started migrating to the Dells, which are far more performant and much more reliable and maintainable (for example, they have built in out of band KVM access). The rest of the costs fluctuates a bit. I had been spending between $600 and $1000 per month on CDN expenses with Bunny.net, but I recently fully migrated to Fastly, which is providing much better pricing (~$50/month). My postgres database performs streaming backups to a Wasabi storage bucket, and that has been costing between $400 and $600 per month. I continue trying to find cheaper alternatives for that, but they never work as reliably as Wasabi, and I am not inclined to take risk with those backups. I have been spending about $200/month on DeepL translation services, though I also recently moved that to a self hosted instance of Libretranslate. I have a bunch of other less expensive costs, like about $65/month on DNS hosting, $50 or so (depending on renewal dates) for domain names, about $100/month for additional backup storage hosted with backblaze (for object storage backups), and so on. Going forward, the costs will be closer to $4000 given the DeepL and CDN changes.
I built this out when we had about 30,000 active users on infosec.exchange and it provided some headroom for more growth, but the opposite has been happening lately and infosec.exchange is now down to less than 13,000 active users.
The breakdown looks like this:
1 - https://infosec.exchange – 12.4k MAU
2 - https://relay.infosec.exchange – 164 connected instances
3 - https://video.infosec.exchange – 41 MAU
4 - https://infosec.press – 6 MAU
5 - https://pixel.infosec.exchange – 114 MAU
6 - https://matrix.infosec.exchange – Not really sure
7 - https://infosec.place – 21 MAU
8 - https://infosec.town – 45 MAU
9 - https://infosec.pub – 369 MAU
10 - https://fedia.io – 597 MAU
11 - https://fedia.social – 46 MAU
12 - https://books.infosec.exchange – 20 MAU
13 - https://meetups.infosec.exchange – minimal
14 - https://infosec.space – 51 MAU
15 - https://convo.casa – 134 MAU
I have been receiving about $5500/month in donations across liberapay, patreon, and ko-fi, but that has been dropping monthly, as the number of accounts declines, and so I’ve started working to optimize costs, and will start looking at moving to smaller servers and more consolidation in the near future.
I asked:
Some people noticed that for $5K one can get a very descent computer. Why not just buy personal hardware and run the instances that way instead of paying for a service?
Jerry answered with this:
I'm not sure those people have experience with running these services. It's really not feasible to run this many services supporting that many accounts on a $5000 PC.
Now, it is true that I'm using more expensive hardware than I need. For one, it's much much easier to maintain, extra capacity/memory means that it runs into problems that require my intervention far less (in fact, it's quite rare that we have any unscheduled issues now), and it's let me absorb some of the shocks that happen when, for example, people bail on instragram and pile onto pixelfed or from reddit to lemmy and mbin.
I started infosec.exchange a bit over 8 years ago and it was a tiny little thing with terrible performance for a few years. I eventually moved it to a larger server and was quite impressed at how nice the experience was when not constrained by capacity. Soon after, twitter melted down. I was quite active in the infosec community on twitter and wanted to give a "soft landing" spot for anyone that wanted to leave, and that's how things really got going. Over time, the community asked me to run different things, like pixelfed, lemmy, iceshrimp, etc, which I obliged, but I felt (and still feel) that it's important to provide a good, reliable, and fast service, and that's honestly been my priority infrastructure-wise. In the early days, I was able to cover any funding gaps because I was earning quite a lot of money, but eventually donations caught up, and now I'm out of work so can't really do that any longer and have to ensure costs are equal to or lower than donations, and that's been working out.
There are a few other reasons I'm using Hetzner: first is scalability and flexibility. Sure, I could buy servers for less than I end up renting them for, but I've turned over the layout of the infrastructure a bunch of times. If I need to add another Dell server, I place an order and shortly I get access to it and receive another $500 charge at the end of the month. When I'm done with it, I just cancel it and I'm no longer charged for it. Next is support - I've had servers die on me (particularly the non-Dell ones) and I cannot tell you how nice it is to just open a support ticket and be running on a brand new copy of the same server (ok, it might actually be used, but you get the point). Last is hosting - I have 9 servers, 5 of them with 10Gbps connection to the internet. That is far from free, even if I own the server.
Would I like to have a 20Gbps connection to my house hosting a cabinet holding $20000 of servers with battery backups, cooling, redundant power, and whatnot? you bet! but that's not really an option for me, so this is the best I've come up with. Are there better options? Probably, but this is how I chose to approach things.
I asked:
I've noticed you list a few options to support your instance with money. But it seems like some people will be rather unwilling to do that kind of thing, for various reasons. For example a lot of people that are very libre-software focused check that even the javascript code in their browser is libre. Mastodon is fine for them. But something like the blob stripe uses for Librepay transactions is not. How could those people, if they wanted to, send you money?
Jerry answered with this:
In 8 years, I think I had one person ask if I would accept crypto currency for donations. I really should set that up, but it's actually not been an issue for me.
I had to clarify something about the whole crypto-currency thing. There is a problem with that approach as well. So I wrote:
It seems like crypto just adds a layer on top of the existing problem. Yes you can send crypto with libre software only, but buying / withdrawing it requires the same non-libre JS as with direct credit card use. But there are ways to send money without the use of computers at all. Like cash, deposit checks, wire transfers, western union and its competitors. And other options that I may not be able to remember from the top of my head.
Jerry answered with this:
No worries - I misunderstood. I have never had anyone ask for a way to donate like that.
I clarified myself further:
It seems like a lot of people not seeing an option to do any of that just automatically assume that they don't want to donate and move on. I wonder how many more will donate if such option will be obviously available. The Free Software Foundation lists a lot of alternative options for such people and they also make sure the JS for transactions is libre: www.fsf.org/about/ways-to-donate
Jerry answered with this:
I'll run a poll to see if there is interest in that - the community here doesn't seem to be shy about asking for alternatives, which is how I ended up with patreon, liberapay, and ko-fi.
I added this question:
If some person has strong believes in libre software and really wants to donate in a method that is not listed. Is there an option for this person to contact you in some way, to figure out some custom way of donating?
Jerry answered with this:
I do list my email address in my profile, and I can objectively say that people have had no problems finding it to ask me all manner of questions 😂
I guess if you do want to donate to Jerry and cannot find a way you would be comfortable with, please contact him @jerry@infosec.exchange . His email as of now is jerry@bell.st .
I asked:
On my website I do this gimmicky thing called petitions. I set a goal ( like say x amount of followers on mastodon ) and I do something when this goal is reached. On the website I use the mastodon API to fetch the followers amount and render a neat progress bar. Is there an API call I can do to get the data needed to render a progress bar of how much money you need to collect this month to have the instance running?
Jerry answered with this:
I think the challenge is how to distribute that across the services. I suspect if I fall below the level I need, I'll look into something like that. Since I had a modestly large follower count, I've been trying to focus on encouraging others to donate to their own respective instances - I think it's important to ensure that this place remains healthy and available as an alternative to corporate social media, and that requires instances to get the money they need. It doesn't do me any good if I have enough to run infosec. but others are having to close up shop. I've even started "adopting" instances rather than seeing them go under due to lack of funding (like convo.casa).
I wrote this in response to that:
I hope I can make a page somewhere, maybe on my website, or somehow on the ActivityPub, which checks for the health of the Fediverse, so to speak. And gives information on who to donate to if the health falls below a comfortable level. Technically speaking, for you, Patreon and Kofi and Librepay should have an API, which you can access with your key. You can then setup a basic JSON dump thingie on one of the servers. That includes only the relevant numbers. And I could use that.
So far no response to that.
I asked:
In one of the replies to the article I wrote you clarified yourself that you don't see how this donations model can be sustainable in the long run if a lot more people join the fediverse. What sort of thing do you think should change for that ( and by "that" I mean everybody uses only fediverse ) to be feasible?
Jerry answered with this:
I'm not sure where this went off the rails. I thought we were talking about a hypothetical future where the fediverse was intended to be THE social media network for the masses - i.e, hundreds of millions or billions of people. The point I was trying to make is we can't expect the scale of infrastructure required to support that many people to be funded through donations by people who generally view that as a "free" service (even if we don't think it really IS free or should be free).
There are other issues packed in there, though. The model we have today where almost all instances are maintained by hobbyists collecting donations to fund hardware/bandwidth/storage. At a scale where there are 100 or 1000 times the people that are currently on the fediverse, we are talking about some pretty large infrastructures that are going to require professional management. I honestly don't see the volunteer admin scaling to that level - I just don't think there are that many people willing to donate their free time, and I also think that the notion that everyone just runs their own instance works, either.
There are technical issues, too, like storage duplication and whatnot - my eyes bleed thinking about the storage bucket(s) and database servers I'd need to hold 100x the amount of data, even for smaller instances, that will equate to large amounts of remote data being stored. I know that can be overcome with protocol and application changes, but then I think we start asymptotically approaching the Bsky/AT model.
I asked:
What are the ratios that one have to expect from such a project? As in how many people use the instance vs how many are donating? And also what is the average donation?
Jerry answered with this:
I'll have to count to give a good answer. My best guess is that there are around 500 donors which would be around 4%. Many seem to donate at the $5/month level, but there are some that do considerably more and some that do $1 (or whatever the minimum is for that donation channel).
So what do we learn from that?
This is a lot of information to digest, so I suppose we should perhaps put that information into an easily digestible list.
- ActivityPub is not well optimized for scalability. Based on my personal adventures with ActivityPub I can tell you that having a lot of people on the fediverse ( even if everybody is running their own instance ) is kind of a problem. Jerry pointed out that a lot of the data on the fediverse is duplicated for easier reach. If you send a reply from mastodon.social to a post on infosec.exchange, infosec.exchange server will need to have a copy of this reply. And if we have billions of people replying to other billions of people all the time, we have to store hundreds of billions of replies on each of the servers. I personally doubt the need to move to Blue-sky's methods of handling this. What we need is temporary storage of data and better ways to fetch histories of data from the original servers. That way the data will be primarily stored only on the servers that made the posts. And others will have the data only while it is relevant. But will remove it to save space when the relevancy decreases. Fetching it back again if the post becomes relevant again. Now if we implement this badly we would remove the storage problem while significantly increasing the bandwidth requirement. ActivityPub is already sending a little bit too many messages. Some software do that better than others. I think the majority of the issue comes from Mastodon itself. And that is something that has to be solved with Mastodon's developers. But with enough smart decision-making in this area this could be solved. We just need to make sure we don't forget about it.
- $5 is a reasonable amount to ask from people. Talking to Jerry I confirmed the belief I had, that on average people send about $5 worth of donations. This is what happens to BTFree right now as well. Donations are coming in, in $5 chunks. So if you want to start making donations to your project, $5 is a reasonable price to ask.
- Only a fraction ( %4 ) of people ever donate. Talking to Jerry we learned that most people who use the fediverse don't donate. Now the reasons could be various. I suspect some of those people just don't like the various options available. And a lot of those people might not even have the funds to begin with. Sometimes you need to worry about bread on your table. So if you want to start donations, be aware you need a lot more people using your service than those you expect to be donating to it.
- Fediverse is barely stable. If Jerry spends about $3.5K while getting from donations about $5.5K which is the numbers we saw, he has about $2K of a nice buffer zone to live off. I don't personally think this is enough money for a comfortable life with a family and stuff. And the costs are rising due to political tensions. We are not yet there to have the problem solved. Remember we want to have a nice enviable life from libre projects. Not one where we barely survive it. Think about the goal in these terms: We want normies to start Fediverse instances and think it's worth it. We are not there yet.
- There is a lack of ways to check Fediverse's health. I was able to ask those question of Jerry, to check the health of Jerry's instances and the answers I've got will be true for this particular moment. But things change. And prices go up. We need to organize some way to know which instances suffer at the moment, so we could direct the funds into places that need them. Perhaps some sort of website that checks the health of the fediverse would be nice to have. For that we have to expose some public API with data about the health.
- The Fediverse is Wavy and unpredictable. Jerry said that at the Twitter exodus ( when Musk bought Twitter ) he had 30K users on infosec.exchange. Which is now down to only 12K. With it the donations also went down. This means a few things. The hype for the fediverse is dying, perhaps because other things like blue sky appeared since then. Yet instances need to have incredible flexibility in order to not be crashed if another influx of users comes by in a near future. Like he pointed out that people run from Instagram and Reddit and other enshitified platforms right now. There are waves in the system. You need to be prepared for that.
- There is a lack of payment options. A lot of people that use the Fediverse use it because it is the only Free Software platform that there is. And those people would be the most reliant of it to keep existing. Because for them to go back to Facebook or Twitter is not even an option. Yet those same people cannot donate because donations require things that are not libre. I really hope that more options will appear to support as many donation channels as possible in as many libre projects as possible. So those people that are the most passionate about the whole thing will be able to support it too.
Happy Hacking!!!
![[thumbnail]](/pictures/thumbs/moria_crashes.png)
![[thumbnail]](/pictures/thumbs/danisrace_2025-03-17.jpg)
![[thumbnail]](https://upload.wikimedia.org/wikipedia/en/thumb/4/4f/The_Audition_%282015_film%29.jpg/250px-The_Audition_%282015_film%29.jpg)
![[thumbnail]](/pictures/thumbs/israel_palestine_hitler.jpg)
![[thumbnail]](https://upload.wikimedia.org/wikipedia/en/8/86/6_Underground_%28film%29.jpg)
![[thumbnail]](/pictures/thumbs/freedom.jpg)
![[thumbnail]](/pictures/thumbs/npcs_stupid.png)
![[thumbnail]](https://upload.wikimedia.org/wikipedia/en/6/65/Queer_%282024%29_film_poster.jpg)
![[thumbnail]](/pictures/strange_town.jpg)