hello云胜

技术与生活

0%

mongo异常退出问题排查

mongo跑了一段时间后莫名其妙的异常退出,在mongo自己的日志里并没有留下任何遗言。

这种情况下,凭借个人的经验,猜测又是系统内存不足,触发OOM-killer,杀死了mongod。

查看 /var/log/messages文件

image-20220520103414628

果然又是OOM的问题

(93条消息) Mongo崩溃crash, 报out of memory的问题分析与解决方案_飞出四季做的茧的博客-CSDN博客_mongodb 崩溃

默认的是,系统内存的一半再减1,我现在的机器是16G的,也就是最高会申请到7G。

太多了,触发oom

设置为2G,观察

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
May 19 18:22:00 pfsmongo1 kernel: NetworkManager invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
May 19 18:22:00 pfsmongo1 kernel: NetworkManager cpuset=/ mems_allowed=0
May 19 18:22:00 pfsmongo1 kernel: CPU: 4 PID: 671 Comm: NetworkManager Kdump: loaded Tainted: G OE ------------ 3.10.0-1160.36.2.el7.x86_64 #1
May 19 18:22:00 pfsmongo1 kernel: Hardware name: OpenStack Foundation OpenStack Nova, BIOS rel-1.10.2-0-g5f4c7b1-20181220_000000-szxrtosci10000 04/01/2014
May 19 18:22:00 pfsmongo1 kernel: Call Trace:
May 19 18:22:00 pfsmongo1 kernel: [<ffffffff94783559>] dump_stack+0x19/0x1b
May 19 18:22:00 pfsmongo1 kernel: [<ffffffff9477e5f8>] dump_header+0x90/0x229
May 19 18:22:00 pfsmongo1 kernel: [<ffffffff94106972>] ? ktime_get_ts64+0x52/0xf0
May 19 18:22:00 pfsmongo1 kernel: [<ffffffff9415dfff>] ? delayacct_end+0x8f/0xb0
May 19 18:22:00 pfsmongo1 kernel: [<ffffffff941c252d>] oom_kill_process+0x2cd/0x490
May 19 18:22:00 pfsmongo1 kernel: [<ffffffff941c1f1d>] ? oom_unkillable_task+0xcd/0x120
May 19 18:22:00 pfsmongo1 kernel: [<ffffffff941c2c1a>] out_of_memory+0x31a/0x500
May 19 18:22:00 pfsmongo1 kernel: [<ffffffff941c9834>] __alloc_pages_nodemask+0xad4/0xbe0
May 19 18:22:00 pfsmongo1 kernel: [<ffffffff94219388>] alloc_pages_current+0x98/0x110
May 19 18:22:00 pfsmongo1 kernel: [<ffffffff941bdfe7>] __page_cache_alloc+0x97/0xb0
May 19 18:22:00 pfsmongo1 kernel: [<ffffffff941c0f80>] filemap_fault+0x270/0x420
May 19 18:22:00 pfsmongo1 kernel: [<ffffffffc03f1756>] ext4_filemap_fault+0x36/0x50 [ext4]
May 19 18:22:00 pfsmongo1 kernel: [<ffffffff941ee78a>] __do_fault.isra.61+0x8a/0x100
May 19 18:22:00 pfsmongo1 kernel: [<ffffffff941eed3c>] do_read_fault.isra.63+0x4c/0x1b0
May 19 18:22:00 pfsmongo1 kernel: [<ffffffff941f6580>] handle_mm_fault+0xa20/0xfb0
May 19 18:22:00 pfsmongo1 kernel: [<ffffffff94790653>] __do_page_fault+0x213/0x500
May 19 18:22:00 pfsmongo1 kernel: [<ffffffff94790a26>] trace_do_page_fault+0x56/0x150
May 19 18:22:00 pfsmongo1 kernel: [<ffffffff9478ffa2>] do_async_page_fault+0x22/0xf0
May 19 18:22:00 pfsmongo1 kernel: [<ffffffff9478c7a8>] async_page_fault+0x28/0x30
May 19 18:22:00 pfsmongo1 kernel: Mem-Info:

NetworkManager invoked oom-killer