On August 1, 2024, the Hemis blockchain experienced a significant incident—a hard fork triggered by a wallet release intended for the launch of our new Gamemaster nodes. As a project committed to transparency and integrity, we are not shying away from discussing this mistake openly. We deeply value our community’s trust and are determined to learn from this incident to prevent future occurrences.
Incident Overview
The root cause of the hard fork was two-fold: the oversight in updating the wallet seed nodes prior to the release, and a deviation from our established internal release management process. These lapses led to the unintended creation of a chain fork, impacting our users and operations on the Xeggex exchange.
Impact on Users and Operations
As a result of the fork:
- A segment of Hemis users found themselves on an incorrect version of the blockchain.
- Deposits and withdrawals were temporarily disabled on Xeggex, causing inconvenience for our traders.
- Fortunately, no user lost tokens on the correct chain. However, the largest stakers on the network faced a “loss” of up to 250 HMS due to unrecoverable staking rewards earned on the forked chain. We are aware of two users who lost this many HMS as a result of the fork.
Immediate Response and Resolution
Upon identifying the fork, we took swift actions to mitigate any further impact:
- We requested Xeggex to halt deposits and withdrawals to minimize the risk of token loss.
- Our team updated the necessary seed nodes and developed a new wallet version. Users were required to download this updated wallet and resync their blockchain to correct their node alignment.
- During our investigation, we discovered that some seed nodes were configured incorrectly, which has since been rectified.
Outcome
The chain fork issue has been fully resolved, and normal trading activities have resumed on Xeggex. Our network is stable, and measures have been put in place to prevent a recurrence of similar issues.
Lessons Learned and Future Improvements
This incident has been a learning experience for our team. We have taken the following steps to enhance our operations and infrastructure:
- Updated Release Process: We have refined our release process and are committed to adhering strictly to it.
- Improved Communication: We are improving our communication protocols for emergency chain events to ensure timely and effective information dissemination.
- Automatic Update Feature: Plans are underway to develop an automatic update feature, which will simplify the process for priority updates.
- Strengthened Internal Communication: We are enhancing our internal communication and ensuring clearer handovers.
Our Commitment Moving Forward
We sincerely apologize to all our users and stakeholders for any inconvenience this incident may have caused. It is our firm resolve to do better. We are taking decisive actions to ensure our technology and processes are robust enough to support the high standards that the Hemis community deserves.
Thank you for your continued support and understanding as we learn and grow from these challenges. Together, we are building a stronger, more secure Hemis blockchain.