We deeply regret to inform you that a critical outage occurred in our storage service, resulting in the loss of customer data. We sincerely apologize for the significant inconvenience and concern this incident has caused our customers. Below, this outline will cover what happened, the technical background of the incident, its causes in the clearest way possible, and our future actions.
| Target node | kita.teracloud.jp node (kita1 storage pool) |
|---|---|
| Affected users | Users who had data stored on the above node (kita). |
| Date of incident | August 19, 2026 (Status: Ongoing) |
This outage is limited to the kita.teracloud.jp node. Customers using other nodes and services have not experienced any data loss and can continue to use them as usual.
You can check whether you are affected by looking at the address bar of the file browser or the WebDAV connection URL.
1. What occurred?
On August 19, 2026, data corruption occurred on the node (kita.teracloud.jp) where customer data was stored.
The storage system consisted of 20 hard drives configured in ZFS RAIDZ3. RAIDZ3 is a highly robust storage configuration that uses triple parity (redundancy information used for data recovery), allowing data to be recovered even if up to three discs fail simultaneously. Under normal circumstances, the failure of several discs would not result in data loss.
However, in this incident, the failure occurred in a way that exceeded the limits of the redundancy provided by RAIDZ3. In addition to one disc becoming completely unresponsive (UNAVAIL), data integrity errors (checksum errors) simultaneously increased across multiple discs.
RAIDZ can recover data from multiple damaged dicks by using the remaining data and parity information to reconstruct the affected data, provided that the damaged areas do not overlap. In this case, however, the damage spread extensively during the recovery process, and the damaged areas overlapped, making recovery using parity information impossible. As a result, most data stored has become unreadable.
2. Progress of the recovery
| Date | Content |
|---|---|
| 8/19 | A malfunction was detected. The impact was anticipated to be limited to a few customers, so the equipment was determined to be a malfunction of the main unit (casing), and the equipment was promptly replaced. |
| 8/20 onward | During the recovery process (resilver), the damage unexpectedly expanded. Checksum errors surged from several thousand to approximately 1.8 million, peaking at several million. The storage was repeatedly suspended during the process. |
| 8/21 | It was confirmed that approximately 4.75 million data files were corrupted. |
| 8/24 onward | A shift to recovering readable data was prioritized. The data was then moved to a different location. |
| 8/26 onward | Despite a relocation process, only 0.2% of total data capacity was recovered. |
※ resilver is a standard reconstruction process to restore the redundancy of lost discs. While it is originally intended for recovery, in this case, the process took an unexpected turn when the extent of the damage was discovered to be expanding as it progressed.
3. Scope of impact
The impact on the affected node (kita.teracloud.jp), based on information available as of September 14, is as follows. The percentages below apply solely to the data within that specific node.
| Category | Percentage (based on capacity/number of files) |
|---|---|
| Lost data | Capacity-based: 99.85% / Number of files-based: 34.2% |
| Recovered data | Capacity-based: 0.2% (actual value due to relocation) |
The loss primarily affected capacity, with approximately one-third of the files remaining intact. The difference indicates that the loss was concentrated on large files. This is to be a result of a structure where, if the higher level of metadata managing the location of data is corrupted, the large group of files attached to it becomes inaccessible, while files remaining on other systems can be read safely.
These figures are determined when corrupted data is actually read. RAIDZ detects data loss on a damaged disc the first time that data is read. Therefore, these percentages may fluctuate until all data has been verified.
4. Regarding the cause
Based on the events that have transpired, we believe the cause was as follows:
Because the damage increased as the recovery process progressed, we believe it was not caused by individual failures of each disc, but rather by a common factor that affected multiple discs simultaneously. We are considering this in terms of two layers: triggering factors and predisposing factors.
Triggering factors
The internal communication paths connecting the discs (SAS paths: HBA, expander, and backplane) failed simultaneously at multiple points. When a path fails, all discs connected to that path become unresponsive. However, a path failure itself does not, in principle, destroy data. This is because ZFS features a transactional design (committing writes in a single batch); even if the connection drops mid-write, only uncommitted operations are lost, while the integrity of already committed data remains intact.
Predisposing factors (underlying factors)
Some discs' firmware may have reported a completed write operation even when the process was actually incomplete. ZFS trusts this "write complete" response from the disc and proceeds to the next step. If this response was incorrect, the underlying assumptions of ZFS data integrity protection would be compromised when the communication path was interrupted or reset. This breakdown could manifest as inconsistencies in the management information that forms the basis of the entire data structure.
Furthermore, we believe that the massive number of errors (~4.75 million) were not caused by individual data blocks becoming corrupted independently. They instead appear to have resulted from inconsistencies in higher-level management information that propagated across a wide range of underlying data. This also explains the previously mentioned discrepancy between 34.2% of files being affected and 99.85% of the storage capacity being affected.
Since the damage continued even after the main unit was replaced, we have concluded, through a process of elimination, that the most likely cause lies with the disc that was not replaced.
5. Recurrence prevention measures
To prevent similar issues from recurring, the following steps will be taken:
Replacement of all affected discs
While no similar data loss has occurred on other nodes, as a preventative measure hard drives with the suspected model numbers used across the service will be replaced. During the replacement process, we will also collect error trend data to further investigate and confirm the issue.
Transition to a Ceph-based architecture
The core issue with this failure was that redundancy was confined to a single device and a single pool, meaning that a complex failure within that system could result in total data loss. Even RAIDZ3's triple parity could not recover the data when the damaged areas overlapped.
While this configuration was the best option available when InfiniCLOUD's (formerly TeraCLOUD) service first launched, we plan to migrate our storage infrastructure to a virtualized infrastructure using Ceph and replicate data across multiple chassis. This will eliminate the dependence on a single device and a single storage pool.
6. Future actions
InfiniCLOUD will:
- Continue to recover and transfer any data that can be retrieved to affected users.
- Provide updates on the situation as they become available.
We sincerely apologize once again for this incident and the inconvenience it has caused.
Posted: September 15, 2026