On the evening of September 22, a user sent an ATOM transfer.
The next morning, the transaction was still marked “Pending.”
The user had not lost their private key, and the wallet showed no abnormal signing activity. Yet multiple public RPC endpoints showed Cosmos Hub stalled at block height 33,086,740.
No new blocks were being produced, so there was nowhere for the transaction to be included.
Only after Cosmos Hub resumed block production roughly a day later did the pending ATOM transfer finally succeed.
For an ordinary user, this may have been one of the clearest possible lessons in how blockchain consensus actually works.
We often say that no central authority can shut down a public blockchain. The reality is more complicated. A sufficiently decentralized blockchain may have no physical “off switch” sitting in a company’s server room, but it can still stop.
The Cosmos Hub halt exposed this normally invisible part of the system to everyday users.
1. Why Did Cosmos Hub Suddenly Stop Producing Blocks?
First, it is important to clarify a common misunderstanding: Cosmos Hub itself was not directly attacked.
The incident began on Neutron.
On September 22, a Neutron governance proposal titled “AIATO: AI Agent Takeover” passed. The attacker exploited chain-level governance permissions and used privileged instructions provided by the wasmd framework to reassign the administrators of application contracts associated with Astroport, Drop, and other protocols to attacker-controlled addresses.
This was not a conventional code vulnerability or protocol bug.
A simple analogy is that each application had its own door lock, but Neutron’s chain-level governance still held a more powerful master key. Once the attacker gained control of the governance outcome, they effectively obtained that master key. They could appoint new administrators, migrate contracts, and ultimately move the assets controlled by those contracts.
Cosmos Hub became involved when some of the stolen assets were transferred across chains.
According to Cosmos Labs’ incident report, the attacker moved assets to several networks before Neutron stopped operating. Approximately 1.7 million ATOM was transferred to Cosmos Hub, where the attacker began exchanging it through cross-chain liquidity.
Cosmos Hub itself had not been exploited, and ordinary Hub users did not lose funds directly because of the Neutron attack.
However, the stolen ATOM had already reached the Hub. To prevent the remaining funds from leaving, a group of Cosmos Hub validators began taking their nodes offline.
By approximately 7:18 p.m. Singapore time on September 22, the validators that had stopped participating represented more than one-third of the network’s total voting power. Cosmos Hub could no longer reach consensus on new blocks and halted at height 33,086,740.
This distinction matters.
No company clicked a “Pause” button, nor did the network first conduct an onchain governance vote to stop. Cosmos Hub halted because enough validators ceased participating in consensus.
What happened next is even more revealing.
Approximately four hours after the halt, validators received a complete recovery plan. It proposed a one-time state change at the halted height that would move the remaining ATOM from the attacker’s address to a multisignature address managed jointly by community validators.
Cosmos Labs then developed, tested, and distributed the Gaia v28.3.0 patch based on the plan accepted by validators.
At the designated recovery height, the patched Gaia software would execute a single state transition, moving 1,227,121 ATOM from the attacker’s address to a 4-of-6 multisig controlled by Nansen, Keplr, Enigma, Silknodes, Kiln, and Polkachu.
By early September 23, validators representing more than 67% of total voting power had confirmed that they had installed Gaia v28.3.0.
Cosmos Hub coordinated its restart at 12:00 UTC that day. Approximately six minutes later, the one-time state change was executed at block height 33,086,741, and the network resumed normal block production.
From halt to recovery, the sequence was clear: validators first caused the network to lose liveness. Then more than two-thirds of the voting power adopted a new state-transition rule, allowing that rule to become part of the canonical state after the restart.
This leads to a deceptively simple question: if Cosmos Hub is decentralized, why can more than one-third of its voting power stop it? And why does restarting it require enough validators to accept and run the same software?
The answer lies in the meaning of consensus itself.
2. Consensus Does Not Mean “The Network Can Never Stop”
One of the most common misunderstandings about blockchain is treating “decentralized” and “incapable of going offline” as synonyms.
Consensus mechanisms actually solve a different problem: how can many independent nodes agree on transaction ordering and the state of a ledger without relying on a central bookkeeper?
Different blockchains answer that question in different ways.
Bitcoin’s best-known mechanism is proof of work. Miners compete to produce blocks using computational power. If two valid branches temporarily emerge, nodes continue building on the chain with the greatest accumulated proof of work.
Bitcoin therefore has no explicit moment at which 67% of participants vote and a block becomes permanently finalized. Instead, it offers probabilistic finality: the more blocks that are built on top of a transaction, the more computationally expensive it becomes to reorganize that transaction out of the chain.
This is why users have traditionally been advised to wait for six confirmations before treating a Bitcoin transaction as sufficiently settled. Even a miner with substantial computing power cannot simply bypass the consensus rules enforced by full nodes.
That does not mean Bitcoin’s state is absolutely unchangeable under every imaginable circumstance. If the broader ecosystem adopted new client software and new consensus rules through a hard fork, a state transition that was invalid under the old rules could theoretically become valid under the new ones.
The real question is who could persuade enough miners, full nodes, exchanges, wallets, and users to accept those new rules together.
In practice, almost no one can.
Bitcoin developers cannot unilaterally determine the network’s consensus rules, nor can miners or exchanges easily do so. The social coordination threshold is extremely high. After Binance lost 7,000 BTC in a 2019 hack, there was brief discussion of contacting major miners to reorganize the chain, but the idea went nowhere.
Ethereum offers another well-known model.
Since moving to proof of stake, Ethereum has used Gasper, a consensus mechanism that combines Casper FFG with the LMD-GHOST fork-choice rule.
In simplified terms, one component determines which chain validators should follow, while the other allows checkpoints to achieve finality.
A checkpoint must receive votes representing at least two-thirds of the total staked ETH before it can progress toward finalization. Conversely, if more than one-third of the stake fails to participate correctly for an extended period, Ethereum may temporarily stop finalizing blocks.
Ethereum also includes an inactivity leak. If the network cannot finalize for long enough, the effective balances of inactive validators gradually decline, eventually allowing the participating majority to regain the two-thirds threshold and restore finality.
Changing an otherwise finalized outcome would again require changes to protocol rules and client software.
The 2016 DAO incident is the classic example. Through a hard fork at block 1,920,000, the Ethereum community executed what the Ethereum Foundation explicitly described as an “irregular state change,” transferring the affected ETH to a recovery contract.
Not everyone accepted that decision. Some miners and community members refused to upgrade and continued maintaining the original state, ultimately creating Ethereum Classic.
Cosmos Hub uses yet another model. Its CometBFT consensus is closer to a conventional Byzantine fault-tolerant (BFT) system.
In simplified terms, a block requires a commit signed by more than two-thirds of the total voting power before it can be accepted.
The advantage is clear finality. Once a block receives sufficient voting power and is committed, users do not need to keep waiting for additional blocks to accumulate probabilistic confidence, as they do with proof of work.
The trade-off is equally direct. If validators representing one-third or more of the voting power stop providing the votes required for a commit, the remaining validators cannot reach the more-than-two-thirds threshold, no matter how hard they try.
Under those conditions, the safest response is to stop producing blocks—as Cosmos Hub did during this incident.
From a distributed-systems perspective, the halt was therefore not mysterious. Once validators controlling enough voting power stopped participating, the consensus protocol followed its rules: it sacrificed availability rather than confirming new blocks without sufficient agreement.
This reflects two concepts that ordinary users often conflate.
- Safety: preventing different nodes from finalizing conflicting states.
- Liveness: ensuring the network can keep progressing and processing new transactions.
When too few nodes participate in a BFT consensus system, temporarily losing liveness may be the price of preserving safety. Put plainly, a decentralized ledger would rather stop than allow different groups of validators to maintain conflicting versions of the truth.
Seen from this perspective, many apparently unrelated incidents in public blockchain history revolve around the same question:
When distributed nodes can no longer agree on the correct state, what should the network do?
3. From Bitcoin to Solana: Where Are the Real Risk Boundaries of Public Blockchains?
Cosmos is not the first network to expose this question.
Bitcoin experienced a classic chain-split incident in 2013.
Bitcoin 0.8 had switched its underlying database from Berkeley DB to LevelDB. When a block containing an unusually large number of transaction inputs appeared, version 0.8 nodes processed it normally, while some older nodes rejected it because of Berkeley DB’s lock limits.
Everyone was still running Bitcoin software, but old and new clients now produced different answers to the same question: was this block valid?
The network split into two chains. The version 0.8 chain temporarily held roughly 60% of the total hash power, preventing the normal proof-of-work process from quickly resolving the split on its own.
Major mining pools ultimately coordinated a downgrade to version 0.7, restoring greater hash power to the chain following the older rules. The network then converged again. The incident was later documented in BIP 50.
Ethereum’s 2016 DAO incident pushed the question one step further.
As discussed above, the Ethereum community adopted a hard fork that executed a special state change at block 1,920,000 and transferred the affected ETH into a recovery contract.
But part of the mining and user community rejected this intervention and continued following the original rules, creating the chain now known as Ethereum Classic.
The DAO fork demonstrated that in an extreme event, social consensus exists alongside consensus encoded in software. If participants cannot reach sufficient agreement, one blockchain can genuinely split into two.
Solana’s 2021 outage demonstrated a different failure mode.
In September of that year, a flood of bot-generated transactions overwhelmed the network. Validator processes exhausted their memory and crashed, leaving the network unable to agree on the current state or confirm new blocks for approximately 17 hours. Validators then coordinated a network restart.
These incidents did not have the same cause.
Bitcoin’s 2013 fork occurred because different client versions began enforcing different validity rules.
Solana’s 2021 outage occurred because large numbers of validators could no longer participate normally in consensus, causing the network to lose liveness.
Ethereum’s DAO fork concerned whether the community should actively alter state by adopting new protocol rules.
The Cosmos Hub incident added another variation. Validators first coordinated to intentionally remove enough participation for the network to lose liveness, preventing stolen assets from continuing to move. A sufficiently large share of voting power then adopted new software and a recovery state, allowing the network to reach agreement again.
Rather than reducing these events to “blockchains can be switched off” or “decentralization is a lie,” it is more useful to recognize a less convenient but more accurate reality:
A consensus mechanism is not a machine that can never fail. It is a decentralized set of rules defining who determines the canonical chain when disagreement occurs, how much participation is required for a state to become final, whether a network should continue or halt during a failure, and what kind of collective action can change the rules governing what happens next.
This leaves ordinary users with a question more important than whether the chain should have been halted.
Final Thoughts
The crypto industry often says, “Not your keys, not your coins.”
That principle remains valid. It describes control over assets: if you hold your own private key, a wallet, exchange, or other third party cannot normally sign a transfer on your behalf.
But it assumes the underlying blockchain can process the signed transaction.
On the day Cosmos Hub stopped producing blocks, users still held their private keys, and their assets did not disappear. Yet even a correctly signed transaction could not be processed because there was no new block to include it.
The recovery process revealed something else. When a sufficient share of consensus participants accepts a new state rule, the state of a particular account can change without a transaction signed by that account’s original private key.
This does not invalidate “Not your keys, not your coins.” It reminds us that sovereignty over private keys and authority over the underlying consensus are not the same thing.
The same distinction applies to wallets.
A wallet can ensure that users retain control of their private keys and signing authority. It can identify chain-level anomalies quickly, display transaction status accurately, maintain redundant RPC and node connections, and verify the final outcome of a transaction after the network recovers.
But a wallet cannot restore consensus on behalf of a blockchain. It cannot guarantee that the underlying network will never be interrupted, nor can it guarantee that the chain’s rules and state will never change through consensus-level decisions.
A mature decentralized system should therefore not pretend that nothing can ever change.
Instead, it should make its imperfect boundaries as clear as possible: who can cause consensus to halt, how much voting power is required, and under what circumstances emergency intervention is permitted.
True decentralization cannot guarantee that a system will never encounter a crisis.
What matters is whether, when a crisis does occur, we can still determine who decided how the ledger would proceed, under which rules, and with how much consensus behind them.