Impala 内存梳理1
原创
Impala 内存
按照底层分类
- JVM Memory
- C++ TcMalloc Memory
- code/stack/.. Untracked Memory
按照对象分类
- Admission_control 单个Cluster/Group 整体资源
- Jvm ...
- BufferPool
- ReservationTracker: BufferPool内存限制 , ExecNode 执行节点在申请内存时,会先在ReservationTracker进行逻辑上的内存社情
- MemTracker 内存统计
Impala-内存限制
BufferPool
BufferPool
Arena 别名参考LevelDB设计, BufferPool 我理解主要是进行按照不同Page大小,缓存不同PerSizeList连续空间..
Page: Logical
Buffer: Phyical
Client(PageHandle) ->(page.buffer)
Page(Pin count ) (Pinned dirty clean)
Unpinned(->splill )
Pinned:
/// -> When pin_in_flight=false, the buffer contains the page's data and the client can
/// read and write to the buffer.
/// -> When pin_in_flight=true, the page's data is in the process of being read from
/// scratch disk into the buffer. Clients will block on the read I/O if they attempt
/// to access the buffer.
Pinned pinned_pages_
Unpinned - Dirty: dirty_unpinned_pages_
Unpinned - Write in flight: write_in_flight_pages_
Unpinned - Clean: When the write to scratch has completed but the page was not evicted. The page is in a clean pages list in a BufferAllocator arena.
Unpinned - Evicted: After a clean page's buffer has been reclaimed. The page is not in any list.ReservationTracker/MemTracker
*Tracker
MemTracker的主要作用是跟踪内存的使用情况
ReservationTracker的主要作用上是从逻辑上控制Impala的资源使用, 代码逻辑上跟MemTracker 是大同小异的
从Process(进程)-Query(查询级别)-FragmentInstance(执行碎片级别)-ExecNode(算子级别) 以一棵树的形态进行内存管理, 在当前子节点内存不同的情况下, 会想父节点申请内存
MemTracker
ReservertionTracker
内存申请过程
由BufferPool Client 申请 主要步骤为
1. client Prepare-> ReservationTracker ->MemTracker 进行资源预申请
2 client Allocate -> BufferAllocator -> (System/Arena->PerSizeList->FreeList) 做实际申请
BufferPool Allocate
步骤一
代办
1 一个Impala SQL 的内存 申请过程
原创声明：本文系作者授权腾讯云开发者社区发表，未经许可，不得转载。
如有侵权，请联系 cloudcommunity@tencent.com 删除。
原创声明：本文系作者授权腾讯云开发者社区发表，未经许可，不得转载。
如有侵权，请联系 cloudcommunity@tencent.com 删除。
评论
登录后参与评论
推荐阅读
目录