비프음이 울리고 파일이 보이지 않아요.
갑자기 나스에서 비프음이 울리더니 공유 폴더에 저장한 파일이 보이지 않습니다. 저장소 관리자에서 확인해 보니 볼륨 1이 충돌했다고 나옵니다. 또한 볼륨 1의 크기와 사용한 용량이 모두 0바이트로 나옵니다. 나스에는 스토리지 풀이 한 개 있습니다. 이 스토리지 풀의 용량을 나눠서 볼륨1과 볼륨 2로 사용하고 있는데요. 볼륨2는 정상적으로 작동하고 있습니다.
디스크 4개를 사용해서 RAID10을 사용하고 있습니다. 할당 상태와 상태 모두 정상으로 나옵니다. 이는 디스크 – 스토리지 풀 – 볼륨의 상관관계에서 디스크와 스토리지 풀은 이상이 없다는 의미입니다.
하나의 스토리지 풀에 생성한 볼륨1과 볼륨2 중에서 볼륨1만 충돌했다면 이는 볼륨1 자체에 문제가 생겼음을 의미합니다.
소요시간 : 3일 어려움 : ★★★★★
데이터 복구 작업을 진행합니다.
볼륨의 크기가 0바이트로 보이는 이유는 볼륨1이 마운트 되지 않기 때문입니다. 리눅스를 설치한 컴퓨터에서는 볼륨1이 마운트됩니다. 하지만 일부 폴더가 보이지 않습니다. 이런 증상의 원인은 파일시스템에 오류가 생겼기 때문으로 생각됩니다. 어쩔수 없이 최후의 방법인 데이터 복구 작업을 통해서 볼륨1의 파일을 복구했습니다.
볼륨1의 파일을 복구한 이후에는 볼륨1을 제거 후 다시 생성합니다. 그리고 공유 폴더를 생성하고 복구한 파일을 원래의 공유 폴더 안에 다시 넣어줍니다.
데이터 복구에 소요되는 시간은 디스크 용량과 개수, 레이드 유형, 데이터 용량에 따라서 결정되는데요. 이번 사례는 4TB 디스크 4개, 레이드 10, 파일의 용량은 5TB 정도이고 소요된 시간은 3일입니다.
복구천사 프로그램이 btrfs 파일시스템도 아주 잘 복구해 줍니다.
궁금하면 ↓↓↓↓
root@DS918p:~# cat /proc/mdstat
Personalities : [raid1] [raid10]
md2 : active raid10 sata3p3[0] sata2p3[3] sata1p3[2] sata4p3[1]
7804393088 blocks super 1.2 64K chunks 2 near-copies [4/4] [UUUU]
md1 : active raid1 sata3p2[0] sata2p2[3] sata1p2[2] sata4p2[1]
2097088 blocks [4/4] [UUUU]
md0 : active raid1 sata3p1[0] sata2p1[3] sata1p1[2] sata4p1[1]
2490176 blocks [4/4] [UUUU]
unused devices:
root@DS918p:~# mdadm -D /dev/md2
/dev/md2:
Version : 1.2
Creation Time : Tue Sep 1 13:02:59 2020
Raid Level : raid10
Array Size : 7804393088 (7442.85 GiB 7991.70 GB)
Used Dev Size : 3902196544 (3721.42 GiB 3995.85 GB)
Raid Devices : 4
Total Devices : 4
Persistence : Superblock is persistent
Update Time : Fri Jan 6 09:09:46 2023
State : clean
Active Devices : 4
Working Devices : 4
Failed Devices : 0
Spare Devices : 0
Layout : near=2
Chunk Size : 64K
Name : NAS:2
UUID : 0cb160f8:90b38940:55e79ee8:5a004888
Events : 62
Number Major Minor RaidDevice State
0 8 35 0 active sync set-A /dev/sata3p3
1 8 51 1 active sync set-B /dev/sata4p3
2 8 3 2 active sync set-A /dev/sata1p3
3 8 19 3 active sync set-B /dev/sata2p3
root@DS918p:~# mdadm -E /dev/sata1p3
/dev/sata1p3:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 0cb160f8:90b38940:55e79ee8:5a004888
Name : NAS:2
Creation Time : Tue Sep 1 13:02:59 2020
Raid Level : raid10
Raid Devices : 4
Avail Dev Size : 7804393120 (3721.42 GiB 3995.85 GB)
Array Size : 7804393088 (7442.85 GiB 7991.70 GB)
Used Dev Size : 7804393088 (3721.42 GiB 3995.85 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
Unused Space : before=1968 sectors, after=32 sectors
State : clean
Device UUID : 9e1aa1a8:44f15335:2d571127:aa71284b
Update Time : Fri Jan 6 09:09:46 2023
Checksum : 6ff85bc4 - correct
Events : 62
Layout : near=2
Chunk Size : 64K
Device Role : Active device 2
Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)
root@DS918p:~# mdadm -E /dev/sata2p3
/dev/sata2p3:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 0cb160f8:90b38940:55e79ee8:5a004888
Name : NAS:2
Creation Time : Tue Sep 1 13:02:59 2020
Raid Level : raid10
Raid Devices : 4
Avail Dev Size : 7804393120 (3721.42 GiB 3995.85 GB)
Array Size : 7804393088 (7442.85 GiB 7991.70 GB)
Used Dev Size : 7804393088 (3721.42 GiB 3995.85 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
Unused Space : before=1968 sectors, after=32 sectors
State : clean
Device UUID : 08af097f:4970fad3:7275955b:2623f65d
Update Time : Fri Jan 6 09:09:46 2023
Checksum : 2c593ef6 - correct
Events : 62
Layout : near=2
Chunk Size : 64K
Device Role : Active device 3
Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)
root@DS918p:~# mdadm -E /dev/sata3p3
/dev/sata3p3:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 0cb160f8:90b38940:55e79ee8:5a004888
Name : NAS:2
Creation Time : Tue Sep 1 13:02:59 2020
Raid Level : raid10
Raid Devices : 4
Avail Dev Size : 7804393120 (3721.42 GiB 3995.85 GB)
Array Size : 7804393088 (7442.85 GiB 7991.70 GB)
Used Dev Size : 7804393088 (3721.42 GiB 3995.85 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
Unused Space : before=1968 sectors, after=32 sectors
State : clean
Device UUID : 1ad4bc4b:a98b5d10:4c37e68c:b6b866f5
Update Time : Fri Jan 6 09:09:46 2023
Checksum : fe30d6ce - correct
Events : 62
Layout : near=2
Chunk Size : 64K
Device Role : Active device 0
Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)
root@DS918p:~# mdadm -E /dev/sata4p3
/dev/sata4p3:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 0cb160f8:90b38940:55e79ee8:5a004888
Name : NAS:2
Creation Time : Tue Sep 1 13:02:59 2020
Raid Level : raid10
Raid Devices : 4
Avail Dev Size : 7804393120 (3721.42 GiB 3995.85 GB)
Array Size : 7804393088 (7442.85 GiB 7991.70 GB)
Used Dev Size : 7804393088 (3721.42 GiB 3995.85 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
Unused Space : before=1968 sectors, after=32 sectors
State : clean
Device UUID : 4910f366:26389037:901a222e:15eb36e5
Update Time : Fri Jan 6 09:09:46 2023
Checksum : d1a5d51e - correct
Events : 62
Layout : near=2
Chunk Size : 64K
Device Role : Active device 1
Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)
root@DS918p:~# smartctl -A /dev/sata1 -d sat
smartctl 6.5 (build date Oct 7 2021) [x86_64-linux-4.4.180+] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF READ SMART DATA SECTION ===
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x000b 100 100 016 Pre-fail Always - 0
2 Throughput_Performance 0x0005 129 129 054 Pre-fail Offline - 104
3 Spin_Up_Time 0x0007 253 253 024 Pre-fail Always - 245 (Average 68)
4 Start_Stop_Count 0x0012 099 099 000 Old_age Always - 6950
5 Reallocated_Sector_Ct 0x0033 100 100 005 Pre-fail Always - 0
7 Seek_Error_Rate 0x000b 100 100 067 Pre-fail Always - 0
8 Seek_Time_Performance 0x0005 128 128 020 Pre-fail Offline - 18
9 Power_On_Hours 0x0012 099 099 000 Old_age Always - 11500
10 Spin_Retry_Count 0x0013 100 100 060 Pre-fail Always - 0
12 Power_Cycle_Count 0x0032 099 099 000 Old_age Always - 6679
192 Power-Off_Retract_Count 0x0032 094 094 000 Old_age Always - 7219
193 Load_Cycle_Count 0x0012 094 094 000 Old_age Always - 7219
194 Temperature_Celsius 0x0002 253 253 000 Old_age Always - 23 (Min/Max 9/46)
196 Reallocated_Event_Count 0x0032 100 100 000 Old_age Always - 0
197 Current_Pending_Sector 0x0022 100 100 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0008 100 100 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x000a 200 200 000 Old_age Always - 0
root@DS918p:~# smartctl -A /dev/sata2 -d sat
smartctl 6.5 (build date Oct 7 2021) [x86_64-linux-4.4.180+] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF READ SMART DATA SECTION ===
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x000b 100 100 016 Pre-fail Always - 0
2 Throughput_Performance 0x0005 129 129 054 Pre-fail Offline - 104
3 Spin_Up_Time 0x0007 253 253 024 Pre-fail Always - 239 (Average 51)
4 Start_Stop_Count 0x0012 099 099 000 Old_age Always - 7009
5 Reallocated_Sector_Ct 0x0033 100 100 005 Pre-fail Always - 0
7 Seek_Error_Rate 0x000b 100 100 067 Pre-fail Always - 0
8 Seek_Time_Performance 0x0005 128 128 020 Pre-fail Offline - 18
9 Power_On_Hours 0x0012 099 099 000 Old_age Always - 11486
10 Spin_Retry_Count 0x0013 100 100 060 Pre-fail Always - 0
12 Power_Cycle_Count 0x0032 099 099 000 Old_age Always - 6739
192 Power-Off_Retract_Count 0x0032 094 094 000 Old_age Always - 7252
193 Load_Cycle_Count 0x0012 094 094 000 Old_age Always - 7252
194 Temperature_Celsius 0x0002 253 253 000 Old_age Always - 23 (Min/Max 9/46)
196 Reallocated_Event_Count 0x0032 100 100 000 Old_age Always - 0
197 Current_Pending_Sector 0x0022 100 100 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0008 100 100 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x000a 200 200 000 Old_age Always - 0
root@DS918p:~# smartctl -A /dev/sata3 -d sat
smartctl 6.5 (build date Oct 7 2021) [x86_64-linux-4.4.180+] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF READ SMART DATA SECTION ===
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x000b 100 100 016 Pre-fail Always - 0
2 Throughput_Performance 0x0005 128 128 054 Pre-fail Offline - 108
3 Spin_Up_Time 0x0007 253 253 024 Pre-fail Always - 118 (Average 146)
4 Start_Stop_Count 0x0012 099 099 000 Old_age Always - 6912
5 Reallocated_Sector_Ct 0x0033 100 100 005 Pre-fail Always - 0
7 Seek_Error_Rate 0x000b 100 100 067 Pre-fail Always - 0
8 Seek_Time_Performance 0x0005 128 128 020 Pre-fail Offline - 18
9 Power_On_Hours 0x0012 099 099 000 Old_age Always - 11500
10 Spin_Retry_Count 0x0013 100 100 060 Pre-fail Always - 0
12 Power_Cycle_Count 0x0032 099 099 000 Old_age Always - 6641
192 Power-Off_Retract_Count 0x0032 095 095 000 Old_age Always - 7156
193 Load_Cycle_Count 0x0012 095 095 000 Old_age Always - 7156
194 Temperature_Celsius 0x0002 253 253 000 Old_age Always - 23 (Min/Max 9/42)
196 Reallocated_Event_Count 0x0032 100 100 000 Old_age Always - 0
197 Current_Pending_Sector 0x0022 100 100 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0008 100 100 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x000a 200 200 000 Old_age Always - 0
root@DS918p:~# smartctl -A /dev/sata4 -d sat
smartctl 6.5 (build date Oct 7 2021) [x86_64-linux-4.4.180+] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF READ SMART DATA SECTION ===
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x000b 100 100 016 Pre-fail Always - 0
2 Throughput_Performance 0x0005 129 129 054 Pre-fail Offline - 104
3 Spin_Up_Time 0x0007 253 253 024 Pre-fail Always - 211 (Average 70)
4 Start_Stop_Count 0x0012 099 099 000 Old_age Always - 6966
5 Reallocated_Sector_Ct 0x0033 100 100 005 Pre-fail Always - 0
7 Seek_Error_Rate 0x000b 100 100 067 Pre-fail Always - 0
8 Seek_Time_Performance 0x0005 128 128 020 Pre-fail Offline - 18
9 Power_On_Hours 0x0012 099 099 000 Old_age Always - 11510
10 Spin_Retry_Count 0x0013 100 100 060 Pre-fail Always - 0
12 Power_Cycle_Count 0x0032 099 099 000 Old_age Always - 6696
192 Power-Off_Retract_Count 0x0032 094 094 000 Old_age Always - 7218
193 Load_Cycle_Count 0x0012 094 094 000 Old_age Always - 7218
194 Temperature_Celsius 0x0002 253 253 000 Old_age Always - 23 (Min/Max 9/46)
196 Reallocated_Event_Count 0x0032 100 100 000 Old_age Always - 0
197 Current_Pending_Sector 0x0022 100 100 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0008 100 100 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x000a 200 200 000 Old_age Always - 0
스토리지 풀에 문제가 없다고 판단하는 근거는 /proc/mdstat 파일의 내용과 mdadm 명령어의 결과에서 특별한 점이 없기 때문입니다. 하드디스크에 문제가 없다고 판단하는 근거는 smartctl 명령어의 결과에서 특별한 점이 없기 때문입니다.
# 나스가 부팅과정에서 볼륨1을 마운트하지 못함. mount 과정에서 생기는 기록. 2023-01-08T10:57:53+09:00 DS918p spacetool[9266]: volume_mount.c:209 Failed to mount '/dev/mapper/cachedev_0' on '/volume1' 2023-01-08T10:57:53+09:00 DS918p spacetool[9266]: volume_mount.c:295 Failed to do volume mount: [/volume1] 2023-01-08T10:57:53+09:00 DS918p kernel: [ 59.208080] BTRFS: open_ctree failed 2023-01-08T10:57:55+09:00 DS918p kernel: [ 61.247355] parent transid verify failed on 1306660798464 wanted 321488 found 321486 2023-01-08T10:57:55+09:00 DS918p kernel: [ 61.338706] parent transid verify failed on 1306660798464 wanted 321488 found 321486 2023-01-08T10:57:55+09:00 DS918p kernel: [ 61.574302] parent transid verify failed on 1306660798464 wanted 321488 found 321486 2023-01-08T10:57:55+09:00 DS918p kernel: [ 61.582978] BTRFS error (device dm-3): BTRFS: dm-3 failed to repair parent transid verify failure on 1306660798464, mirror = 1 2023-01-08T10:57:56+09:00 DS918p kernel: [ 61.606043] parent transid verify failed on 1306660798464 wanted 321488 found 321486 2023-01-08T10:57:56+09:00 DS918p kernel: [ 61.924474] BTRFS error (device dm-3): BTRFS: dm-3 failed to repair parent transid verify failure on 1306660798464, mirror = 2 2023-01-08T10:57:56+09:00 DS918p kernel: [ 61.939290] BTRFS error (device dm-3): failed to load usrquota subtree 257, ret=-5 2023-01-08T10:57:56+09:00 DS918p kernel: [ 61.947740] BTRFS error (device dm-3): usrquota disabled due to faield to load tree 2023-01-08T10:57:57+09:00 DS918p kernel: [ 63.531197] btree_readpage_end_io_hook: 2 callbacks suppressed # 부팅 후에 수동으로 mount 명령어를 사용해도 비슷한 기록이 생김. 2023-01-08T11:11:17+09:00 DS918p kernel: [ 863.834395] btree_readpage_end_io_hook: 8 callbacks suppressed 2023-01-08T11:11:17+09:00 DS918p kernel: [ 863.840926] parent transid verify failed on 1306660798464 wanted 321488 found 321486 2023-01-08T11:11:17+09:00 DS918p kernel: [ 863.932305] parent transid verify failed on 1306660798464 wanted 321488 found 321486 2023-01-08T11:11:17+09:00 DS918p kernel: [ 864.167567] parent transid verify failed on 1306660798464 wanted 321488 found 321486 2023-01-08T11:11:17+09:00 DS918p kernel: [ 864.176213] BTRFS error (device dm-3): BTRFS: dm-3 failed to repair parent transid verify failure on 1306660798464, mirror = 1 2023-01-08T11:11:17+09:00 DS918p kernel: [ 864.199769] parent transid verify failed on 1306660798464 wanted 321488 found 321486 2023-01-08T11:11:17+09:00 DS918p kernel: [ 864.290975] parent transid verify failed on 1306660798464 wanted 321488 found 321486 2023-01-08T11:11:18+09:00 DS918p kernel: [ 864.526057] parent transid verify failed on 1306660798464 wanted 321488 found 321486 2023-01-08T11:11:18+09:00 DS918p kernel: [ 864.534695] BTRFS error (device dm-3): BTRFS: dm-3 failed to repair parent transid verify failure on 1306660798464, mirror = 2 2023-01-08T11:11:18+09:00 DS918p kernel: [ 864.549557] BTRFS error (device dm-3): failed to load usrquota subtree 257, ret=-5 2023-01-08T11:11:18+09:00 DS918p kernel: [ 864.558006] BTRFS error (device dm-3): usrquota disabled due to faield to load tree 2023-01-08T11:11:18+09:00 DS918p kernel: [ 864.586864] BTRFS: open_ctree failed
볼륨1의 파일시스템에 오류가 있다고 판단하는 근거는 mount 명령어를 사용하면 여러가지 오류메시지가 발생하기 때문입니다.
root@PC-MORANG-debian:/mount-point/md127$ ls -al 폴더_A/자식폴더/
ls: cannot access 폴더_A/자식폴더/폴더_C: 입력/출력 오류
ls: cannot access 폴더_A/자식폴더/폴더_D: 입력/출력 오류
ls: cannot access 폴더_A/자식폴더/폴더 E: 입력/출력 오류
합계 0
drwxrwxrwx 1 1027 users 298 1월 2 13:58 .
drwxr-xr-x 1 root root 824 1월 2 13:58 ..
drwxrwxrwx 1 1028 users 398 9월 7 10:20 폴더_B
d????????? ? ? ? ? ? 폴더_C
d????????? ? ? ? ? ? 폴더_D
d????????? ? ? ? ? ? 폴더_E
drwxrwxrwx 1 1027 users 1328 10월 13 09:38 폴더_F
drwxrwxrwx 1 1027 users 480 2월 18 2022 폴더_G
root@PC-MORANG-debian:/mount-point/md127$ 리눅스를 설치한 컴퓨터에는 볼륨1이 마운트 됩니다. 하지만 일부 폴더에서 이름만 정상적으로 보이고 권한, 소유자, 그룹, 수정일 등은 물음표로 표기되는 증상이 발생합니다. 또한 그 폴더에 액세스하면 시놀로지 나스에서발생한 오류와 똑같은 ‘BTRFS error (device dm-1): parent transid verify failed on 1306660798464 wanted 321488 found 321486’ 내용이 kern.log에 쌓입니다.
