Hi guys
Lately, we are experiencing several Out-of-Memory situations on our SAP Landscape, after it was migrated to SAP HANA Database
It could occur randomly in ERP, NW PO or BW, since all of them were migrated from SQL Server to HANA Database
Basically, SAP Instance freezes, and only option is to restart it.
Under alert it shows
preprocessor on host uyclsxxx port 31002 was restarted at 2000-01-01 00:00:00.0000000.
webdispatcher on host uyclsxxx port 31006 was restarted at 2000-01-01 00:00:00.0000000.
Memory Overview seems to be normal and there are no peaks thay might hint Memory is below required
I was able to obtain some diagnostic files and all seems to be related to HANA Server unable to allocate more memory.
Nameserver, Indexserver and webdispatcher Out-Of-Memory dumps all shows the same information.
Any idea on how to troubleshoot these out-ot-memory errors? Bear in mind it could occur at any time, even on non-working hours or weekend when there are no activity on the SAP System
[MEMORY_OOM] Information about current out of memory situation: (2016-05-04 21:38:05 562 Local)
OUT OF MEMORY occurred.
Host: uycls157.mdp.local
Executable: hdbnameserver
PID: 22961
Failed to allocate 32816 byte.
Out of memory reasons:
!:! <1> MemoryPool::allocate(big/huge) failure.
/sapmnt/ld7272/a/HDB/jenkins_prod/workspace/FA_CO_LIN64GCC47HAPPY_rel_fa~newdb100_rel/sys/src/Basis/MemoryManager/impl/MemoryPool.cpp (1416)
!:! <2> BigBlockAllocator failure, no precharge.
/sapmnt/ld7272/a/HDB/jenkins_prod/workspace/FA_CO_LIN64GCC47HAPPY_rel_fa~newdb100_rel/sys/src/Basis/MemoryManager/impl/MemoryPool.cpp (2085)
!:! <3> IPMM early exit.
/sapmnt/ld7272/a/HDB/jenkins_prod/workspace/FA_CO_LIN64GCC47HAPPY_rel_fa~newdb100_rel/sys/src/Basis/MemoryManager/impl/GlobalMemoryHandler.cpp (6866)
!:! <4> MemoryPool::allocate(big/huge) failure.
/sapmnt/ld7272/a/HDB/jenkins_prod/workspace/FA_CO_LIN64GCC47HAPPY_rel_fa~newdb100_rel/sys/src/Basis/MemoryManager/impl/MemoryPool.cpp (1416)
!:! <5> BigBlockAllocator failure, no precharge.
/sapmnt/ld7272/a/HDB/jenkins_prod/workspace/FA_CO_LIN64GCC47HAPPY_rel_fa~newdb100_rel/sys/src/Basis/MemoryManager/impl/MemoryPool.cpp (2085)
!:! <6> IPMM early exit.
/sapmnt/ld7272/a/HDB/jenkins_prod/workspace/FA_CO_LIN64GCC47HAPPY_rel_fa~newdb100_rel/sys/src/Basis/MemoryManager/impl/GlobalMemoryHandler.cpp (6866)
At most one OOM dump is written in approximately 86400s.
[OS_MEMORY] Operating system information about memory: (2016-05-04 21:38:26 465 Local)
/proc/meminfo:
MemTotal: 154912776 kB
MemFree: 1644660 kB
Buffers: 155040 kB
Cached: 37670000 kB
SwapCached: 0 kB
Active: 120563596 kB
Inactive: 30454160 kB
Active(anon): 113194280 kB
Inactive(anon): 22856080 kB
Active(file): 7369316 kB
Inactive(file): 7598080 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 6291448 kB
SwapFree: 6291448 kB
Dirty: 8836 kB
Writeback: 0 kB
AnonPages: 113192916 kB
Mapped: 23249932 kB
Shmem: 22857644 kB
Slab: 749560 kB
SReclaimable: 624456 kB
SUnreclaim: 125104 kB
KernelStack: 5416 kB
PageTables: 417168 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 83747836 kB
Committed_AS: 136213928 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 542572 kB
VmallocChunk: 34359179772 kB
HardwareCorrupted: 0 kB
AnonHugePages: 0 kB
HugePages_Total: 0
HugePag
/proc/vmstat:
nr_free_pages 411165
nr_inactive_anon 5714020
nr_active_anon 28298570
nr_inactive_file 1899520
nr_active_file 1842329
nr_unevictable 0
nr_mlock 0
nr_anon_pages 28298229
nr_mapped 5812483
nr_file_pages 9456260
nr_dirty 2209
nr_writeback 0
nr_slab_reclaimable 156114
nr_slab_unreclaimable 31276
nr_page_table_pages 104292
nr_kernel_stack 677
nr_unstable 0
nr_bounce 0
nr_vmscan_write 433
nr_writeback_temp 0
nr_isolated_anon 0
nr_isolated_file 0
nr_shmem 5714411
numa_hit 3058504450
numa_miss 0
numa_foreign 0
numa_interleave 114388
numa_local 3058504450
numa_other 0
nr_anon_transparent_hugepages 0
pgpgin 2381614114
pgpgout 3146033750
pswpin 433
pswpout 433
pgalloc_dma 2
pgalloc_dma32 155436
pgalloc_normal 3065907965
pgalloc_movable 0
pgfree 3066476428
pgactivate 9823911
pgdeactivate 46976
pgfault 3140334036
pgmajfault 1262
pgrefill_dma 0
pgrefill_dma32 46976
pgrefill_normal 0
pgrefill_movable 0
pgsteal_dma 0
pgsteal_dma32 433
pgsteal_normal 2869225
pgsteal_movable 0
pgscan_kswapd_dma 0
pgscan_kswapd_dma32 2720
pgsc
/proc/sysvipc/shm:
key shmid perms size cpid lpid nattch uid gid cuid cgid atime dtime ctime rss swap
0 67010560 740 1024 22935 22943 1 1000 79 1000 79 1461791037 1461791037 1461791037 0 0
11001 67043329 600 268435456 22943 24417 7 1000 79 1000 79 1461792167 1461792167 1461791037 268435456 0
1446068590 67076098 660 117 22961 24369 6 1000 500 1000 79 1461792151 1461791363 1461791039 4096 0
351291229 67108867 660 1060911 22961 24369 24 1000 500 1000 79 1461792151 1461791363 1461791039 1060864 0
785145360 67141636 660 108 22961 24369 6 1000 500 1000 79 1461792151
1461791363 1461791040 4096 0