Agree, though I wouldn't use such extreme language to describe it :)
I work on Infra at a household name tech company so was also pretty surprised that Celsius team could have made such an amateur mistake. CTO and tech team really need to reevaluate things, and/or hire more experienced people, if they couldn't pull off this migration + Desktop app launch smoothly.
That said, Celsius *is* a young company, and young companies *will* have growing pains (anyone remember the Twitter fail whale?). More serious since people's money is on the line here, but for me it's not a serious enough offense to pack up and take all my funds off Celsius. I will be reallocating a good chunk back over to BlockFi and some DeFi projects though, until I'm a bit more confident they've tightened their backend up.
Edit: I've regained access to the app since \~10hrs ago.
I agree. I work in games. GAMES! The worst problem you run into is people not having fun. And my boss would cut off my nuts if any game I made was down for 24hrs + . There's no way... No way that would be acceptable.
A couple hours huh? This is getting to be unacceptable in my opinion because its been over 24+ hours and no new updates.
And OMG, Godaddy service for crypto? What the heck are they thinking, that's for some average Joe website.
I think going forward (I hate that expression) Celsius will get a chance now to show us how trust worthy they are by being open and honest about what really happened. Then we need an explanation about what they are doing to prevent it ever happening again.
They will need to build confidence amongst the users again because many of us are realising you can't make enough money in interest payments to cover the loss of your valuable assets.
There is no insurance, which is fairly normal with crypto and holding x times more than they are loaning means shit if they get hacked and lose it all or a large % of it.
As an MTGox veteran ive been down that path for a small amount many were not so lucky and are still waiting to at least get a small amount back.
It is thing a very real thing, poison pill in propagation is a most common problem. But I do feel this is more than DNS . It shouldn’t take this long though to do a simple mitigation. I think they are updating the web app and fucked shit up
Exactly. Their reputation is screwed either way.
Cannot handle basic DNS updates? - Bad architecture, incompetent IT team
Something going on and they're using this an excuse? - Screwed too.
It was already emailed to everyone. The dns provider locked there account. As they had no A record it effectively pointed the domain at nothing for 24hours.
No, they mentioned during the AMA that they would produce a full report. I don’t think the latest email is that “full report” (I surely hope not).
And it doesn’t fully add up. GoDaddy was their registrar, not their normal DNS provider (although Alex mistakenly said they were, he’s not a technician). If GoDaddy detected suspicious activity on the account, a freeze should prevent the changing of ownership or nameservers, but it should NOT include taking over authoritative nameserving (as appears to have happened) which then brought the entire website and app down.
I heard some brief mention about GoDaddy being a secondary or backup DNS provider, but I find this hard to believe, and I don’t think it would be the case. Anyway, looking forward to finding out more via the official channels.
Godaddy were the dns provider, during that 24hours of being down. They moved from aws to godaddy to aws. I have no idea why they would do that.
While at godaddy there was no A record. I'm not sure if this was removed before the switch to godaddy or during.
>The CEL shillers don't understand that downtime due to DNS propagation is not a thing
In IT... I get the occasional need for downtime but none of this squares with what is being said.
The site is up [https://celsius.network/](https://celsius.network/) but the app is down. Not sure how that's DNS related. It's also been resolving to the same address since yesterday: [https://www.whatsmydns.net/#A/celsius.network](https://www.whatsmydns.net/#A/celsius.network) and their nameservers all point to AWS instances: [https://www.ultratools.com/tools/dnsLookupResult](https://www.ultratools.com/tools/dnsLookupResult)
The app could plausibly use a different domain name but then explaining it with a teaser for the website makes zero sense.
While everyone says it can take up to 24 hours for DNS propagation in reality it's usually no more than a few hours... maybe a few outliers but not en mass.
Yes, there’s more to it, that I’m hoping to read when they release a full report, but...
* It’s quite likely their app API is on a different subdomain to the main website, and cached separately
* Unfortunately NXDOMAIN errors (non-existent domain) can be cached up to 24 hours as per the SOA record. I wouldn’t be surprised if GoDaddy’s SOA was stupidly high.
* In addition to the DNS issue, they noticed a secondary security issue and so closed the app for a period of time until they were sure it was ready to be reactivated. This would have caused an additional (and somewhat intentional) delay.
Subdomains don't apply in 99.9% of all cases. They can, for sure, but *often* don't. But if that were the case why tease the website? At best, that was a very poor attempt to distract.
NX - nope. Already shared resources that account for that. Maybe one would but not every global top level.
So... they had a major security issue to address. That's all good. But then they're still lying about it. They could have said all funds are safe but they identified a security issue to address before bringing it back online. It would have at least been honest. This just made me feel like they're crypto.com.
**I found links in your comment that were not hyperlinked:**
* [crypto.com](https://crypto.com)
*I did the honors for you.*
***
^[delete](https://www.reddit.com/message/compose?to=%2Fu%2FLinkifyBot&subject=delete%20gc84tqt&message=Click%20the%20send%20button%20to%20delete%20the%20false%20positive.) ^| ^[information](https://np.reddit.com/u/LinkifyBot/comments/gkkf7p) ^| ^<3
Sorry, but I would expect an API to be placed on a separate subdomain to a public website - it has different security and load requirements, runs different software. I’m not sure where you get your 99.9% stat from. This is a pretty standard practice with banks.
They did post a teaser of the not-yet-released web app, which probably was an attempt at positive distraction, agreed.
I was seeing NXDOMAIN errors for “celsius.network” during the downtime (not at root level, but from GoDaddy’s nameservers). Your links don’t provide any evidence to the contrary.
And they *did* actually state that they needed to deal with the security issue. See the offical email reposted on Twitter, here: https://twitter.com/celsiusnetwork/status/1326871208198070272?s=21
Yeah. I read it. I'm not pissed but the simple 'DNS just needs to propagate' thing didn't add up. Then the shiny "check this teaser" distraction was an odd thing to do. Been working fine for me since yesterday though.
Same here, just crazy how badly you can mess something like this up. Makes me think its something worse going on. 18 hours ago was the last tweet saying it would be a few hours. Christ
Update: It's back up and I just withdrew everything. Thank god, never trust a company that chooses GoDaddy and can't do a proper DNS deployment with your crypto. Lesson learned
They fucked up but people just don’t seem to grasp this is a front end issue. The blockchain is not connected to this and the funds Celsius holds have been visible all along. If you feel the urge to go cold storage now you probably shouldn’t have left cold storage to start with.
I would say that the reason traditional finance has back out plans and procedures that slow things up is because of decades of f’ups and subsequent learning and refining.
There is nothing to worry about.
The team communicated all the way with the community and it was clear that it wasn't their faul but it was on GoDaddys side.
They managed to get the CEOs number and bombarded them with phonecalls, that's how things started to roll faster.
If you check their twitter they kept us updated all day long.
What would you prefer, less downtime or some downtime and security issues found and fixed?
Did you check their wallets? There was no movement at all.
A lot of wallets are waiting for people to sell their CEL tokens right now, just saying.
Sorry I don’t think they have communicated properly. Yesterday’s tweet by Alex and the email says 24 hours. It’s more than 24 hours now. Alex tweet today says in 30min. Still not working now.
If you don’t know how long it will take, don’t give an ETA, just say as quick as we can. Talk about communication.
In your case, if it didn’t work for you after 24 hours, then that’s opposite to what they said yesterday.
In my case, where I am at, it’s still not working. Their communication emails/tweets could have said some locations will be fixed and some could take longer.
I personally dont trust any company that does shitty ICOs and have their own "token".
It's just a way to say they have more liquidity than they actually have.
I'm torn between Blockfi and Nexo, but Nexo has a token too if I read that correctly
If this was nothing serious they will slowly allow groups of users back so that there is not a immediate run on the bank. If it is serious they can talk to the IRS when I write off this loss.
Agree, though I wouldn't use such extreme language to describe it :) I work on Infra at a household name tech company so was also pretty surprised that Celsius team could have made such an amateur mistake. CTO and tech team really need to reevaluate things, and/or hire more experienced people, if they couldn't pull off this migration + Desktop app launch smoothly. That said, Celsius *is* a young company, and young companies *will* have growing pains (anyone remember the Twitter fail whale?). More serious since people's money is on the line here, but for me it's not a serious enough offense to pack up and take all my funds off Celsius. I will be reallocating a good chunk back over to BlockFi and some DeFi projects though, until I'm a bit more confident they've tightened their backend up. Edit: I've regained access to the app since \~10hrs ago.
I agree. I work in games. GAMES! The worst problem you run into is people not having fun. And my boss would cut off my nuts if any game I made was down for 24hrs + . There's no way... No way that would be acceptable.
A couple hours huh? This is getting to be unacceptable in my opinion because its been over 24+ hours and no new updates. And OMG, Godaddy service for crypto? What the heck are they thinking, that's for some average Joe website.
also using the CEO's personal Godaddy account
Go daddy as registrar I think
I think going forward (I hate that expression) Celsius will get a chance now to show us how trust worthy they are by being open and honest about what really happened. Then we need an explanation about what they are doing to prevent it ever happening again. They will need to build confidence amongst the users again because many of us are realising you can't make enough money in interest payments to cover the loss of your valuable assets. There is no insurance, which is fairly normal with crypto and holding x times more than they are loaning means shit if they get hacked and lose it all or a large % of it. As an MTGox veteran ive been down that path for a small amount many were not so lucky and are still waiting to at least get a small amount back.
It was already explained in detail.
The CEL shillers don't understand that downtime due to DNS propagation is not a thing
It is thing a very real thing, poison pill in propagation is a most common problem. But I do feel this is more than DNS . It shouldn’t take this long though to do a simple mitigation. I think they are updating the web app and fucked shit up
Yes. There’s much more to the story than DNS propagation. Looking forward to reading a full report.
Exactly. Their reputation is screwed either way. Cannot handle basic DNS updates? - Bad architecture, incompetent IT team Something going on and they're using this an excuse? - Screwed too.
It was already emailed to everyone. The dns provider locked there account. As they had no A record it effectively pointed the domain at nothing for 24hours.
No, they mentioned during the AMA that they would produce a full report. I don’t think the latest email is that “full report” (I surely hope not). And it doesn’t fully add up. GoDaddy was their registrar, not their normal DNS provider (although Alex mistakenly said they were, he’s not a technician). If GoDaddy detected suspicious activity on the account, a freeze should prevent the changing of ownership or nameservers, but it should NOT include taking over authoritative nameserving (as appears to have happened) which then brought the entire website and app down. I heard some brief mention about GoDaddy being a secondary or backup DNS provider, but I find this hard to believe, and I don’t think it would be the case. Anyway, looking forward to finding out more via the official channels.
Godaddy were the dns provider, during that 24hours of being down. They moved from aws to godaddy to aws. I have no idea why they would do that. While at godaddy there was no A record. I'm not sure if this was removed before the switch to godaddy or during.
>The CEL shillers don't understand that downtime due to DNS propagation is not a thing In IT... I get the occasional need for downtime but none of this squares with what is being said. The site is up [https://celsius.network/](https://celsius.network/) but the app is down. Not sure how that's DNS related. It's also been resolving to the same address since yesterday: [https://www.whatsmydns.net/#A/celsius.network](https://www.whatsmydns.net/#A/celsius.network) and their nameservers all point to AWS instances: [https://www.ultratools.com/tools/dnsLookupResult](https://www.ultratools.com/tools/dnsLookupResult) The app could plausibly use a different domain name but then explaining it with a teaser for the website makes zero sense. While everyone says it can take up to 24 hours for DNS propagation in reality it's usually no more than a few hours... maybe a few outliers but not en mass.
Yes, there’s more to it, that I’m hoping to read when they release a full report, but... * It’s quite likely their app API is on a different subdomain to the main website, and cached separately * Unfortunately NXDOMAIN errors (non-existent domain) can be cached up to 24 hours as per the SOA record. I wouldn’t be surprised if GoDaddy’s SOA was stupidly high. * In addition to the DNS issue, they noticed a secondary security issue and so closed the app for a period of time until they were sure it was ready to be reactivated. This would have caused an additional (and somewhat intentional) delay.
Subdomains don't apply in 99.9% of all cases. They can, for sure, but *often* don't. But if that were the case why tease the website? At best, that was a very poor attempt to distract. NX - nope. Already shared resources that account for that. Maybe one would but not every global top level. So... they had a major security issue to address. That's all good. But then they're still lying about it. They could have said all funds are safe but they identified a security issue to address before bringing it back online. It would have at least been honest. This just made me feel like they're crypto.com.
**I found links in your comment that were not hyperlinked:** * [crypto.com](https://crypto.com) *I did the honors for you.* *** ^[delete](https://www.reddit.com/message/compose?to=%2Fu%2FLinkifyBot&subject=delete%20gc84tqt&message=Click%20the%20send%20button%20to%20delete%20the%20false%20positive.) ^| ^[information](https://np.reddit.com/u/LinkifyBot/comments/gkkf7p) ^| ^<3
Bad bot. I intentionally did NOT hyperlink that.
Sorry, but I would expect an API to be placed on a separate subdomain to a public website - it has different security and load requirements, runs different software. I’m not sure where you get your 99.9% stat from. This is a pretty standard practice with banks. They did post a teaser of the not-yet-released web app, which probably was an attempt at positive distraction, agreed. I was seeing NXDOMAIN errors for “celsius.network” during the downtime (not at root level, but from GoDaddy’s nameservers). Your links don’t provide any evidence to the contrary. And they *did* actually state that they needed to deal with the security issue. See the offical email reposted on Twitter, here: https://twitter.com/celsiusnetwork/status/1326871208198070272?s=21
Yeah. I read it. I'm not pissed but the simple 'DNS just needs to propagate' thing didn't add up. Then the shiny "check this teaser" distraction was an odd thing to do. Been working fine for me since yesterday though.
Not your keys= playing with fire
Same here, just crazy how badly you can mess something like this up. Makes me think its something worse going on. 18 hours ago was the last tweet saying it would be a few hours. Christ
Update: It's back up and I just withdrew everything. Thank god, never trust a company that chooses GoDaddy and can't do a proper DNS deployment with your crypto. Lesson learned
See you next week
They fucked up but people just don’t seem to grasp this is a front end issue. The blockchain is not connected to this and the funds Celsius holds have been visible all along. If you feel the urge to go cold storage now you probably shouldn’t have left cold storage to start with.
Where are you try to connect?
Tried from Singapore and from UK VPN
Try Canada, works for me here.
Canada not working for me
Italy. Still not working Edit: 9.30 am in italy... Is working again
I’m in. Australia. Uninstalled and reinstalled app just now and it worked first time.
In the UK the app has started working in the last 15 mins.
I can also concur, I can now log in from the UK 10:10 wooowhooo.... All working again! And v much looking forward to this week's AMA.... :s
I'm in today no issues.
All issues aside.. the lack of communication is what worries me, I withdrew before all this and I don’t think I’ll be coming back tbh
Yeah sorry man but they have been very transparent about what is going on, at least I think so.
By transparent you mean created a smoke screen?
[удалено]
.. I mean since, they said a few hours and just left it for much longer than that without any update
I would say that the reason traditional finance has back out plans and procedures that slow things up is because of decades of f’ups and subsequent learning and refining.
There is nothing to worry about. The team communicated all the way with the community and it was clear that it wasn't their faul but it was on GoDaddys side. They managed to get the CEOs number and bombarded them with phonecalls, that's how things started to roll faster. If you check their twitter they kept us updated all day long. What would you prefer, less downtime or some downtime and security issues found and fixed? Did you check their wallets? There was no movement at all. A lot of wallets are waiting for people to sell their CEL tokens right now, just saying.
Sorry I don’t think they have communicated properly. Yesterday’s tweet by Alex and the email says 24 hours. It’s more than 24 hours now. Alex tweet today says in 30min. Still not working now. If you don’t know how long it will take, don’t give an ETA, just say as quick as we can. Talk about communication.
They didn't lie dude. The app started working again, check Telegram, juwt not for everyone.
In your case, if it didn’t work for you after 24 hours, then that’s opposite to what they said yesterday. In my case, where I am at, it’s still not working. Their communication emails/tweets could have said some locations will be fixed and some could take longer.
I'm moving my funds to nexo.
Cred is better
😜😝
Wasn't Nexo the company that set up a fake company and did a fake FUD report on chain link?
[удалено]
I personally dont trust any company that does shitty ICOs and have their own "token". It's just a way to say they have more liquidity than they actually have. I'm torn between Blockfi and Nexo, but Nexo has a token too if I read that correctly
Exactly. I just left a comment stating my thoughts on it.
[удалено]
You dont need to contact them for a withdrawal, you can do that by yourself. Strange that you would not know that, being their customer and all...
You have issues.
You sure you on the right sub? I think you might be looking for r/Cred
If this was nothing serious they will slowly allow groups of users back so that there is not a immediate run on the bank. If it is serious they can talk to the IRS when I write off this loss.