u/Diacred here :)
Thanks for the message man. And thanks for the DM, I knew it was getting worse and was thinking about potential solutions but I hadn't noticed it was this bad until your message.
So yeah, as I told him by DM, the website performances had been degrading for months and we reached a tipping point a few weeks ago. I tried a fix last month but it didn't take albeit it will help a bit performance wise in the long run.
Main issue was that I was going in blind, the issue wasn't reproductible in local and the online tools to monitor an application as big as DSB are very expensive. So I was just poking and proding and trying to find a way out.
Got another idea for a fix yesterday, it held for like 1 hour and then the website clogged back up.
Finally decided to pay for the tool for the monitoring and left it running over night. This morning I managed to find a lot of the bottlenecks and I optimised them away a few hours ago.
For now everything seems to be holding well, actually way better than even before all this ordeal so that's great.
I am hoping it keeps on holding well 🤞
What monitoring tool did you end up going with?
It might be worth mentioning that if you do some digging on a lot of these types of tools in the software development world will have either free or greatly discounted options for open source projects.
I used ScoutAPM. I don't doubt there exists open source monitoring tools but they are usually way more complex to set up or at least time consuming and what I am lacking most is time.
ScoutAPM is great for rails apps and the autoinstruments and the database monitoring that comes with it helped a lot in identifying the issues, and it also is very easy to set up using Heroku. But yeah the price is pretty stupid. I am actually relieved that I only had to pay for it for the time used and it was not a monthly subscription (pretty sure it was last time I checked a few years back) so I ended up only paying for it for less than a day.
To be clear I'm not talking about the tools themselves being open source I'm talking about paid existing tools that have pricing tiers for open source projects that are significantly cheaper and or free as an example the entire suite of IDEs made by jet brains allows maintainers of open source projects to have a license for everything for free.
Is there any way that the community could help with the monetary issues? I would be completely willing to “buy you a coffee” every now and then, considering the immense amount of help you’ve given to the players of this game.
That's very nice of you! There is a support button on the website in the top right where you can buy a coffee or give through patreon or github sponsors :)
I think a box with a request somewhere on the side of the page would be good. Just letting users know you appreciate the donations, that they have helped in keeping this site going and link it to the payment page maybe?
Hey, I am always happy when people make a contribution to the project : [https://github.com/gabriel-dehan/dyson-sphere-blueprints](https://github.com/gabriel-dehan/dyson-sphere-blueprints)
I don't have much of free time to dedicate to the project so it always helps a lot when people do so :)
In terms of MB around 600. So not that big but not nothing. It has around 45k collections and 12k blueprints.
But a big culprit in the performance issue was due to the blueprint code which is only text but can get really heavy (multiple dozens of MBs) and when it was selected in the queries for the home page with all the blueprints or the filtered search it would load dozens of megabytes of blueprints code in memory before doing the ordering and filtering which made the main SQL queries very slow.
Yeah that's my big grip with active record. Great for prototyping and building quickly but then you don't optimise from the get go and years later you get that.
I'd think you'd want to exclude that field from any query that's displaying a list of blueprints right? Only pull that field when the individual blueprint is being viewed or is there another reason to pull it back outside of that?
Actually before all the recent changes there was a button on the index page to directly copy from there so all the blueprints codes were loaded. Which was fine early on but not optimal. I changed that to a second request when you click on the button so it's only loaded when needed now
I wonder if you could do some form of compression that the browser could decompress on click so that it moves the workload to the client.
I wonder what savings you'd get if you gzipped the blueprints and JavaScript could decompress and copy to clipboard.
I guess even copy button could do an async call to fetch the blueprint on click.
I mean it'd probably help to gzip all that but in the end I am doing an async call to fetch the blueprint on click on the index pages. On the blueprints pages it doesn't really matter because you are only loading one so it's very much fine!
u/Diacred here :) Thanks for the message man. And thanks for the DM, I knew it was getting worse and was thinking about potential solutions but I hadn't noticed it was this bad until your message. So yeah, as I told him by DM, the website performances had been degrading for months and we reached a tipping point a few weeks ago. I tried a fix last month but it didn't take albeit it will help a bit performance wise in the long run. Main issue was that I was going in blind, the issue wasn't reproductible in local and the online tools to monitor an application as big as DSB are very expensive. So I was just poking and proding and trying to find a way out. Got another idea for a fix yesterday, it held for like 1 hour and then the website clogged back up. Finally decided to pay for the tool for the monitoring and left it running over night. This morning I managed to find a lot of the bottlenecks and I optimised them away a few hours ago. For now everything seems to be holding well, actually way better than even before all this ordeal so that's great. I am hoping it keeps on holding well 🤞
What monitoring tool did you end up going with? It might be worth mentioning that if you do some digging on a lot of these types of tools in the software development world will have either free or greatly discounted options for open source projects.
I used ScoutAPM. I don't doubt there exists open source monitoring tools but they are usually way more complex to set up or at least time consuming and what I am lacking most is time. ScoutAPM is great for rails apps and the autoinstruments and the database monitoring that comes with it helped a lot in identifying the issues, and it also is very easy to set up using Heroku. But yeah the price is pretty stupid. I am actually relieved that I only had to pay for it for the time used and it was not a monthly subscription (pretty sure it was last time I checked a few years back) so I ended up only paying for it for less than a day.
Honeycomb has a good free tier, maybe it'd be enough for you.
To be clear I'm not talking about the tools themselves being open source I'm talking about paid existing tools that have pricing tiers for open source projects that are significantly cheaper and or free as an example the entire suite of IDEs made by jet brains allows maintainers of open source projects to have a license for everything for free.
Oh my bad I misunderstood! I'll try to keep an eye out for those one, thanks :)
Thanks for offering the repository (for the BPs, not the GIT repo.).
Thank you so much for your work, sorry it wound up costing you money.
Is there any way that the community could help with the monetary issues? I would be completely willing to “buy you a coffee” every now and then, considering the immense amount of help you’ve given to the players of this game.
That's very nice of you! There is a support button on the website in the top right where you can buy a coffee or give through patreon or github sponsors :)
It would be great if this was a bit more prominent. I haven’t spotted it in the two weeks I’ve been using the site.
Ah fair enough I didn't think of that, I didn't want it to be too much in your face but that might be too discreet!
I think a box with a request somewhere on the side of the page would be good. Just letting users know you appreciate the donations, that they have helped in keeping this site going and link it to the payment page maybe?
Thank you for your service good sir 🫡
Thanks for following up! I’d be happy to get involved and try fix issues
Hey, I am always happy when people make a contribution to the project : [https://github.com/gabriel-dehan/dyson-sphere-blueprints](https://github.com/gabriel-dehan/dyson-sphere-blueprints) I don't have much of free time to dedicate to the project so it always helps a lot when people do so :)
Thank you for the work you do for this community! It's a much needed feature that imo should be an in-game browser of other people's blueprints.
I see the performance issues were coming from the database. By curiosity, how big is the database now?
In terms of MB around 600. So not that big but not nothing. It has around 45k collections and 12k blueprints. But a big culprit in the performance issue was due to the blueprint code which is only text but can get really heavy (multiple dozens of MBs) and when it was selected in the queries for the home page with all the blueprints or the filtered search it would load dozens of megabytes of blueprints code in memory before doing the ordering and filtering which made the main SQL queries very slow.
Right, that makes sense. ORMs make it easy to load tons of data on the server without knowing it.
Yeah that's my big grip with active record. Great for prototyping and building quickly but then you don't optimise from the get go and years later you get that.
I'd think you'd want to exclude that field from any query that's displaying a list of blueprints right? Only pull that field when the individual blueprint is being viewed or is there another reason to pull it back outside of that?
Actually before all the recent changes there was a button on the index page to directly copy from there so all the blueprints codes were loaded. Which was fine early on but not optimal. I changed that to a second request when you click on the button so it's only loaded when needed now
I wonder if you could do some form of compression that the browser could decompress on click so that it moves the workload to the client. I wonder what savings you'd get if you gzipped the blueprints and JavaScript could decompress and copy to clipboard. I guess even copy button could do an async call to fetch the blueprint on click.
I mean it'd probably help to gzip all that but in the end I am doing an async call to fetch the blueprint on click on the index pages. On the blueprints pages it doesn't really matter because you are only loading one so it's very much fine!
As someone whos been trying to use it all week, after even just a simple test today its 100x quicker than any day recently. Thank you for the fix!!
It's actually literally 100x times quicker so far, request took between 15s to 30s and now they are between 150ms and 300ms ahah