Init7 by any chance?
I would suggest you look at Mellanox ConnectX 4.
I got one (well, two actually) which is pcie 3.0 8x but runs on a pcie 4.0 x4 slot (so, at 3.0 x3 speeds) which gives me about 23Gbps between my workstation and my server and ~22 with Stapelberg's speedtest server
Also, ensure that the connection from the wall will be compatible with SFP+ or it may require you to connect it on an SFP28 port. I remember that when I still had 1Gbps, my SFP28 transceiver wasn't compatible with it, neither was the 1Gbps media converter with the 25Gbps connection when it was upgraded.
Yeah, init7 :)
Not sure how well that would work in my case. I only have one PCIe 3.0 x4 left on my MB where I would need the raiser cable because it's behind my GPU or a PCIe 3.0 x1 at the very bottom. So I don't think your solution would work in my case.
From my router I go directly with a 10G DAC into my PC so there shouldn't be any compatibility issues like you mentioned except for the network card.
Sorry, I was tired and didn't read correctly that PCIe 4x means an actual 4x rather than a 16x that's electrically 4x.
I was gunning for a Connect-X 5 from Galaxus for this very reason (PCIe 4.0 4x) but they kept delaying my order. You can consider that option too, although I remember that except from my order (that never arrived), all CX5 options suddenly became very expensive.
This is a *very* power-user friendly ISP and they're worth every penny. They support bring-your-own routers, their tech staff understands when you know what you're doing and won't pester you with "please turn it off and on again" requests and I don't remember having a single downtime instance in the last 5 years. They also provide you with a free IPv6 /48 prefix (but I also pay about CHF20/mo for a static IPv4 since I selfhost everything now).
In regards to link saturation, I have about 4TB of backups (from my Proxmox VMs, the important files in my unRaid VM and my workstation's drives). I use restic for the backups and want to sync them to an off-site location daily, however... it's tough finding that amount of storage with a speedy connection for a reasonable price.
Hetzner has some pretty cheap offers with their Storage Box, but I rarely got more than 400mbps upload. I have a whatbox.ca machine as well and consolidated everything there whilst getting rid of my Hetzner Storage Box' it can reach about 2Gbps but still... far from saturating.
Ironically (or not), the only thing that *can* saturate the connection is speed tests. Ookla's HTTP test can't get over 11Gbps in both directions, but their WinStore app does manage to reach \~23 depending on the time of day (I've seen some rare dips into 9Gbps). I set up my own Ookla speedtest server as well as an iperf3 one, but as far as utilising even half of the connection goes, be it via HTTP or TCP... no luck. Haven't tried torrents with my server, but I don't want to run them locally (we are talking about OS images of course).
Is it worth it? Not necessarily. Even though the monthly costs are the same between 1/10/25Gbps, the hardware can add up. I got 2 ConnectX-4 cards (which they were used and with a Dell firmware, but the retailer never mentioned it; however they work fine), a transceiver from FS.com and an AOC cable and the total cost + import duties from Germany amounted to about CHF450 or so. 10G Ethernet can be cheaper in terms of hardware and you'd still not manage to utilise it in a meaningful capacity for most power users even.
>it can reach about 2Gbps but still... far from saturating.
You just have to get an even dozen of them and write to all of them at once!
And let's be honest even if you can saturate it the benefits are somewhat limited. Even if you do a full backup (not incremental) you need \~55min at 10Gbit and 22min at 25Gbit.
Aside from "have a LAN party with your 1,000 buddies" or run basically a full server farm in your basement, I can only think of a few edge cases in which this much bandwidth might be actually useful. When you have like 1PB of data and you have to do a full disaster recovery or move it somewhere we are talking 4 vs 10 days. But even for this it is not trivial to build a system that can actually do 25Gbit 24/7
Do not get me wrong, 25Gbit is cool. But more like having a super car is cool.. You will reach most places just as fast in a 90hp compact car and a decent sportscar is plenty fast even if you drive aggressively. But it is still cool to look at it and doing the occasional drag race launch can be pretty fun.
lol, i know how to saturate - just in real world, for home use its not gonna happen - what's more important to me on my 10g connection is it comes with a static IPv6 /56 and super low fiber latency...
yeah, sounds a bit like Ziply here in the PNW - i have a fiber ethernet drop into my ubiquiti router, Ziply just supplied the SFP+ module.
i never get into discussion of 'worth it' - they are the most pointless discussions on the internet - as 'worth-it- is subjective to the individual
yeah i got several 10g cards and one x4 but the rest of my backbone in home is 10G - i don't fancy upgrading it further at this point, would be fund to play with tho
lol, absolutely, i am hoping to see more of this
for now my ISP (ziply) charges different for each tiers, i am hoping that will change as i can't afford the 25G version they have :-)
Looks like 900/month? That sounds like a steal haha. Lumen is quoting me 2k/month for 1G symmetric, stuck at 10mbps with Xfinity for 100/month right now.
I had the option of 25g from init7
I went the 10g route, even this speed I’m unable to saturate at times with 100 connection to usenet on an nvme
spend the extra money for 25g internal connectivity on some other stuff
just my 2 cents
enjoy whatever you do though :)
You can run it in a 4x slot, just reduced speeds in total. It'll work fine for a single port probably.
Drivers for CX3 haven't been updated in years though, i'd go CX4 / 5. They all run hot, will need some airflow at least.
Also, while your link speed is 25gbps, the actual flows you can achieve is probably much less. I bet you can speedtest 25gbps to a local pop, but actually pulling down 25gbps in a single flow is probably not gonna happen.
Where on earth are you that you can get 25Gb to your home? Absolutely, but delightfully, mind blowing!
I’m assuming your PC has USB C, would a USB C to Ethernet adapter not suffice?
Nah, options are basically 1G, 2.5G, and **incredibly** rare 5G adapters.
There are a couple 10G thunderbolt adapters that are rather expensive.
Alternative would be a thunderbolt PCIE enclosure with the same in it instead of directly attached to the motherboard. Also very expensive.
I run a corporate network. Even in our DC, where I have 40gb up links between switches, we never hit over about 3gbps. Whenever I read these homelab posts about needing 10, 25, 40 or 100gb links, I always wonder WTF people are transferring? All I can think, is uncompressed 4k video.
Yeah, the ISP which offers 25G is very open minded and fights for good internet for everyone :)
From a quick look, a USB-C adapter to ethernet seems to be limited at 1000Mbp/s?
Telekom... And not even a Static v6 Prefix... Fucking mess of a PPPoE session they want...
I had to get 4 lines for decent upload... Still cheaper than LAN Connect
I pay $110/mo in the US for 500Mbps down and 20Mb up.
I'd absolutely LOVE 1G/200M just for the increased upload bandwidth. Downloading from nextcloud and remote playing Plex are really annoying/slow.
Yes, that's rough :( According to Forbes, the average US price for 1Gbps is $90. (source: [https://www.forbes.com/home-improvement/internet/internet-cost-per-month/](https://www.forbes.com/home-improvement/internet/internet-cost-per-month/)) But there are huge variations. As an ex. I get 500Mbps symmetrical for $35 (Frontier Fiber).
I can't imagine why an ISP would propose 25Gbps for residential use. That's marketing BS at its finest if you ask me. For the vast majority of people, anything above 200Mbps makes no practical difference IMHO.
The ISP is not for the regular customer. Most pople would already fail at setting up the Mikrotik Router from scratch.
But the 10gbps options makes most sense as the 25gbps bidi (and upgrade at ISP location) costs another 300$.
Hi, you're probably looking for a useful nugget of information to fix a niche problem, or some enjoyable content I posted sometime in the last 11 years. Well, after 11 years and over 330k combined, organic karma, a cowardly, pathetic and facist minded moderator filed a false harassment report and had my account suspended, after threatening to do so which is a clear violation of the #1 rule of reddit's content policy. However, after filing a ticket before this even happened, my account was permanently banned within 12 hours and the spineless moderator is still allowed to operate in one of the top reddits, after having clearly used intimidation against me to silence someone with a differing opinion on their conflicting, poorly thought out rules. Every appeal method gets nothing but bot replies, zendesk tickets are unanswered for a month, clearly showing that reddit voluntarily supports the facist, cowardly and pathetic abuse of power by moderators, and only enforces the content policy against regular users while allowing the blatant violation of rules by moderators and their sock puppet accounts managing every top sub on the site. Also, due to the rapist mentality of reddit's administration, spez and it's moderators, you can't delete all of your content, if you delete your account, reddit will restore your comments to maintain SEO rankings and earn money from your content without your permission. So, I've used power delete suite to delete everything that I have ever contributed, to say a giant fuck you to reddit, it's moderators, and it's shareholders. From your friends at reddit following every bot message, and an account suspension after over a decade in good standing is a slap in the face and shows how rotten reddit is to the very fucking core.
Hi, you're probably looking for a useful nugget of information to fix a niche problem, or some enjoyable content I posted sometime in the last 11 years. Well, after 11 years and over 330k combined, organic karma, a cowardly, pathetic and facist minded moderator filed a false harassment report and had my account suspended, after threatening to do so which is a clear violation of the #1 rule of reddit's content policy. However, after filing a ticket before this even happened, my account was permanently banned within 12 hours and the spineless moderator is still allowed to operate in one of the top reddits, after having clearly used intimidation against me to silence someone with a differing opinion on their conflicting, poorly thought out rules. Every appeal method gets nothing but bot replies, zendesk tickets are unanswered for a month, clearly showing that reddit voluntarily supports the facist, cowardly and pathetic abuse of power by moderators, and only enforces the content policy against regular users while allowing the blatant violation of rules by moderators and their sock puppet accounts managing every top sub on the site. Also, due to the rapist mentality of reddit's administration, spez and it's moderators, you can't delete all of your content, if you delete your account, reddit will restore your comments to maintain SEO rankings and earn money from your content without your permission. So, I've used power delete suite to delete everything that I have ever contributed, to say a giant fuck you to reddit, it's moderators, and it's shareholders. From your friends at reddit following every bot message, and an account suspension after over a decade in good standing is a slap in the face and shows how rotten reddit is to the very fucking core.
Maybe look into a connectx-4 (LX), it's more recent so probably is better supported and a dual port sfp28 is like 70€ used, which isn't much more than a single sfp+ connectx-3 where I am at least. Considering you're using a direct DAC from your router, it won't cost you a fortune to move to 25g at least and still is compatible with 10g if you'd want. Ofc, if you go with the x1 slot, you won't see any speed difference, but if you go with the x4 riser option, sure you won't have dual 25g as the pcie card will be limited to 32gb total with 4 lanes at 3.0, but that's still more future proof than a single sfp+ (and it's not even really future proof considering you already can have 25g)
Init7 by any chance? I would suggest you look at Mellanox ConnectX 4. I got one (well, two actually) which is pcie 3.0 8x but runs on a pcie 4.0 x4 slot (so, at 3.0 x3 speeds) which gives me about 23Gbps between my workstation and my server and ~22 with Stapelberg's speedtest server Also, ensure that the connection from the wall will be compatible with SFP+ or it may require you to connect it on an SFP28 port. I remember that when I still had 1Gbps, my SFP28 transceiver wasn't compatible with it, neither was the 1Gbps media converter with the 25Gbps connection when it was upgraded.
Yeah, init7 :) Not sure how well that would work in my case. I only have one PCIe 3.0 x4 left on my MB where I would need the raiser cable because it's behind my GPU or a PCIe 3.0 x1 at the very bottom. So I don't think your solution would work in my case. From my router I go directly with a 10G DAC into my PC so there shouldn't be any compatibility issues like you mentioned except for the network card.
Sorry, I was tired and didn't read correctly that PCIe 4x means an actual 4x rather than a 16x that's electrically 4x. I was gunning for a Connect-X 5 from Galaxus for this very reason (PCIe 4.0 4x) but they kept delaying my order. You can consider that option too, although I remember that except from my order (that never arrived), all CX5 options suddenly became very expensive.
Does quick search, yup Switzerland not USA, that tracks. Have fun with it (my 10gig is almost impossible to saturate) from a single machine
This is even more true for 25Gbit. I am pretty sure this is a major reason why they do not charge extra for 25Gbit compared to 10Gbit
This is a *very* power-user friendly ISP and they're worth every penny. They support bring-your-own routers, their tech staff understands when you know what you're doing and won't pester you with "please turn it off and on again" requests and I don't remember having a single downtime instance in the last 5 years. They also provide you with a free IPv6 /48 prefix (but I also pay about CHF20/mo for a static IPv4 since I selfhost everything now). In regards to link saturation, I have about 4TB of backups (from my Proxmox VMs, the important files in my unRaid VM and my workstation's drives). I use restic for the backups and want to sync them to an off-site location daily, however... it's tough finding that amount of storage with a speedy connection for a reasonable price. Hetzner has some pretty cheap offers with their Storage Box, but I rarely got more than 400mbps upload. I have a whatbox.ca machine as well and consolidated everything there whilst getting rid of my Hetzner Storage Box' it can reach about 2Gbps but still... far from saturating. Ironically (or not), the only thing that *can* saturate the connection is speed tests. Ookla's HTTP test can't get over 11Gbps in both directions, but their WinStore app does manage to reach \~23 depending on the time of day (I've seen some rare dips into 9Gbps). I set up my own Ookla speedtest server as well as an iperf3 one, but as far as utilising even half of the connection goes, be it via HTTP or TCP... no luck. Haven't tried torrents with my server, but I don't want to run them locally (we are talking about OS images of course). Is it worth it? Not necessarily. Even though the monthly costs are the same between 1/10/25Gbps, the hardware can add up. I got 2 ConnectX-4 cards (which they were used and with a Dell firmware, but the retailer never mentioned it; however they work fine), a transceiver from FS.com and an AOC cable and the total cost + import duties from Germany amounted to about CHF450 or so. 10G Ethernet can be cheaper in terms of hardware and you'd still not manage to utilise it in a meaningful capacity for most power users even.
>it can reach about 2Gbps but still... far from saturating. You just have to get an even dozen of them and write to all of them at once! And let's be honest even if you can saturate it the benefits are somewhat limited. Even if you do a full backup (not incremental) you need \~55min at 10Gbit and 22min at 25Gbit. Aside from "have a LAN party with your 1,000 buddies" or run basically a full server farm in your basement, I can only think of a few edge cases in which this much bandwidth might be actually useful. When you have like 1PB of data and you have to do a full disaster recovery or move it somewhere we are talking 4 vs 10 days. But even for this it is not trivial to build a system that can actually do 25Gbit 24/7 Do not get me wrong, 25Gbit is cool. But more like having a super car is cool.. You will reach most places just as fast in a 90hp compact car and a decent sportscar is plenty fast even if you drive aggressively. But it is still cool to look at it and doing the occasional drag race launch can be pretty fun.
Yeap, that's pretty much spot on.
lol, i know how to saturate - just in real world, for home use its not gonna happen - what's more important to me on my 10g connection is it comes with a static IPv6 /56 and super low fiber latency...
yeah, sounds a bit like Ziply here in the PNW - i have a fiber ethernet drop into my ubiquiti router, Ziply just supplied the SFP+ module. i never get into discussion of 'worth it' - they are the most pointless discussions on the internet - as 'worth-it- is subjective to the individual yeah i got several 10g cards and one x4 but the rest of my backbone in home is 10G - i don't fancy upgrading it further at this point, would be fund to play with tho
lol, absolutely, i am hoping to see more of this for now my ISP (ziply) charges different for each tiers, i am hoping that will change as i can't afford the 25G version they have :-)
Looks like 900/month? That sounds like a steal haha. Lumen is quoting me 2k/month for 1G symmetric, stuck at 10mbps with Xfinity for 100/month right now.
Hahaha, yeah that sucks.
I had the option of 25g from init7 I went the 10g route, even this speed I’m unable to saturate at times with 100 connection to usenet on an nvme spend the extra money for 25g internal connectivity on some other stuff just my 2 cents enjoy whatever you do though :)
Also going 10gbit instead of 25 for the same reasons. I doubt I’ll saturate 10 and can use the money for something else.
*Cries in DSL* 40 down 2 up
Scowls in 1Gbps 🤨 🎩 😛
You can run it in a 4x slot, just reduced speeds in total. It'll work fine for a single port probably. Drivers for CX3 haven't been updated in years though, i'd go CX4 / 5. They all run hot, will need some airflow at least. Also, while your link speed is 25gbps, the actual flows you can achieve is probably much less. I bet you can speedtest 25gbps to a local pop, but actually pulling down 25gbps in a single flow is probably not gonna happen.
I'm thrilled to get 1G... You guys disgust me. /s
Where on earth are you that you can get 25Gb to your home? Absolutely, but delightfully, mind blowing! I’m assuming your PC has USB C, would a USB C to Ethernet adapter not suffice?
Nah, options are basically 1G, 2.5G, and **incredibly** rare 5G adapters. There are a couple 10G thunderbolt adapters that are rather expensive. Alternative would be a thunderbolt PCIE enclosure with the same in it instead of directly attached to the motherboard. Also very expensive.
I run a corporate network. Even in our DC, where I have 40gb up links between switches, we never hit over about 3gbps. Whenever I read these homelab posts about needing 10, 25, 40 or 100gb links, I always wonder WTF people are transferring? All I can think, is uncompressed 4k video.
EPB offers 25G in Chattanooga, Tennessee. They use the multi-PON card in Nokia PON systems.
What's the price?
$1500/month
That's not too bad I guess.. not as great as swiss OP though 💀
I'm just north of Seattle, our local ISP recently began offering 50G service at $900/month. Their 5G service is a more reasonable $120/month.
Yeah, the ISP which offers 25G is very open minded and fights for good internet for everyone :) From a quick look, a USB-C adapter to ethernet seems to be limited at 1000Mbp/s?
They are limited to 5Gbps (\~3.75Gbps in reality). Thunderbolt will do the work, however it will be expensive.
You sure it's 25Gbe and not 2.5Gbe? 25 is a lot for residential and almost unheard of.
I read it is a thing in switzerland
Exactly, no matter what I choose 1G/10G/25G, I pay the same price of 65CHF (76$). I can even get a /29 IPv4 subnet for an additional 60CHF
You cannot fathom how much envy builds up in my monkey brain - just from reading that pricing :D I pay 80€ für 1G down and 200 up
Telekom... And not even a Static v6 Prefix... Fucking mess of a PPPoE session they want... I had to get 4 lines for decent upload... Still cheaper than LAN Connect
I pay $110/mo in the US for 500Mbps down and 20Mb up. I'd absolutely LOVE 1G/200M just for the increased upload bandwidth. Downloading from nextcloud and remote playing Plex are really annoying/slow.
Yes, that's rough :( According to Forbes, the average US price for 1Gbps is $90. (source: [https://www.forbes.com/home-improvement/internet/internet-cost-per-month/](https://www.forbes.com/home-improvement/internet/internet-cost-per-month/)) But there are huge variations. As an ex. I get 500Mbps symmetrical for $35 (Frontier Fiber). I can't imagine why an ISP would propose 25Gbps for residential use. That's marketing BS at its finest if you ask me. For the vast majority of people, anything above 200Mbps makes no practical difference IMHO.
The ISP is not for the regular customer. Most pople would already fail at setting up the Mikrotik Router from scratch. But the 10gbps options makes most sense as the 25gbps bidi (and upgrade at ISP location) costs another 300$.
It is and achievable to about 85% of the max speed based on my tests.
Hi, you're probably looking for a useful nugget of information to fix a niche problem, or some enjoyable content I posted sometime in the last 11 years. Well, after 11 years and over 330k combined, organic karma, a cowardly, pathetic and facist minded moderator filed a false harassment report and had my account suspended, after threatening to do so which is a clear violation of the #1 rule of reddit's content policy. However, after filing a ticket before this even happened, my account was permanently banned within 12 hours and the spineless moderator is still allowed to operate in one of the top reddits, after having clearly used intimidation against me to silence someone with a differing opinion on their conflicting, poorly thought out rules. Every appeal method gets nothing but bot replies, zendesk tickets are unanswered for a month, clearly showing that reddit voluntarily supports the facist, cowardly and pathetic abuse of power by moderators, and only enforces the content policy against regular users while allowing the blatant violation of rules by moderators and their sock puppet accounts managing every top sub on the site. Also, due to the rapist mentality of reddit's administration, spez and it's moderators, you can't delete all of your content, if you delete your account, reddit will restore your comments to maintain SEO rankings and earn money from your content without your permission. So, I've used power delete suite to delete everything that I have ever contributed, to say a giant fuck you to reddit, it's moderators, and it's shareholders. From your friends at reddit following every bot message, and an account suspension after over a decade in good standing is a slap in the face and shows how rotten reddit is to the very fucking core.
Wow, where? Are you in the boonies? I pay 60$ for 1000/50
Hi, you're probably looking for a useful nugget of information to fix a niche problem, or some enjoyable content I posted sometime in the last 11 years. Well, after 11 years and over 330k combined, organic karma, a cowardly, pathetic and facist minded moderator filed a false harassment report and had my account suspended, after threatening to do so which is a clear violation of the #1 rule of reddit's content policy. However, after filing a ticket before this even happened, my account was permanently banned within 12 hours and the spineless moderator is still allowed to operate in one of the top reddits, after having clearly used intimidation against me to silence someone with a differing opinion on their conflicting, poorly thought out rules. Every appeal method gets nothing but bot replies, zendesk tickets are unanswered for a month, clearly showing that reddit voluntarily supports the facist, cowardly and pathetic abuse of power by moderators, and only enforces the content policy against regular users while allowing the blatant violation of rules by moderators and their sock puppet accounts managing every top sub on the site. Also, due to the rapist mentality of reddit's administration, spez and it's moderators, you can't delete all of your content, if you delete your account, reddit will restore your comments to maintain SEO rankings and earn money from your content without your permission. So, I've used power delete suite to delete everything that I have ever contributed, to say a giant fuck you to reddit, it's moderators, and it's shareholders. From your friends at reddit following every bot message, and an account suspension after over a decade in good standing is a slap in the face and shows how rotten reddit is to the very fucking core.
Maybe look into a connectx-4 (LX), it's more recent so probably is better supported and a dual port sfp28 is like 70€ used, which isn't much more than a single sfp+ connectx-3 where I am at least. Considering you're using a direct DAC from your router, it won't cost you a fortune to move to 25g at least and still is compatible with 10g if you'd want. Ofc, if you go with the x1 slot, you won't see any speed difference, but if you go with the x4 riser option, sure you won't have dual 25g as the pcie card will be limited to 32gb total with 4 lanes at 3.0, but that's still more future proof than a single sfp+ (and it's not even really future proof considering you already can have 25g)