前言
最近在看我之前的写的一篇关于人脸landmark的文章,里面有提到OpenVINO自带模型人脸的35个点位,有人问我这个landmark检测有什么用,我斗胆抛砖引玉一下,做了个简单的自动戴口罩的AR演示。
模型介绍
用到两个OpenVINO模型,分别是:
face-detection-0204 人脸检测
facial-landmarks-35-adas-0002 landmark检测35点
face-detection-0204模型的输入与输出格式如下:
输入:1x3x448x448, BGR顺序
输出:1x1x200x7
facial-landmarks-35-adas-0002模型的输入与输出格式如下:
输入:1x3x60x60
输出:1x70
其中70是35个点的xy坐标,取值范围在0~1之间
代码实现
代码实现部分首先检测人脸,然后截取人脸的ROI区域,检测35点landmark坐标,选择(18,34)或者(19、33)两个点位作为起始位置,与对应口罩图象对齐,高度选编码26的点,这样计算X与Y方向的放缩比率,完成对齐,然后贴图口罩到指定的人脸区域。
其中人脸检测的代码如下:
# Read IR
net = ie.read_network(model=face_xml, weights=face_bin)
input_blob = next(iter(net.input_info))
out_blob = next(iter(net.outputs))
# 输入设置
n, c, h, w = net.input_info[input_blob].input_data.shape
# 设备关联推理创建
exec_net = ie.load_network(network=net, device_name="CPU")
# 加载landmark模型并设置
landmark_net = ie.read_network(model=landmark_35_xml, weights=landmark_35_bin)
landmark_input_blob = next(iter(landmark_net.input_info))
landmark_out_blob = next(iter(landmark_net.outputs))
# 输入设置
pn, pc, ph, pw = landmark_net.input_info[landmark_input_blob].input_data.shape
# 设备关联推理创建
landmark_exec_net = ie.load_network(network=landmark_net, device_name="CPU")
src = cv.imread("D:/1.jpg")
image = cv.resize(src, (w, h))
image = image.transpose(2, 0, 1)
# 推理
prob = exec_net.infer(inputs={input_blob: [image]})
# 后处理
ih, iw, ic = src.shape
res = prob[out_blob]
if res.ndim == 4: # SSD
for obj in res[0][0]:
if obj[2] > 0.5:
xmin = int(obj[3] * iw) - 15
ymin = int(obj[4] * ih) - 15
xmax = int(obj[5] * iw) + 15
ymax = int(obj[6] * ih) + 15
if xmin < 0:
xmin = 0
if ymin < 0:
ymin = 0
if xmax >= iw:
xmax = iw - 1
if ymax >= ih:
ymax = ih - 1
roi = src[ymin:ymax, xmin:xmax, :]
infer_landmark(roi, landmark_exec_net, landmark_input_blob, landmark_out_blob, pw, ph)
# cv.rectangle(src, (xmin, ymin), (xmax, ymax), (0, 255, 255), 2, 8)
# cv.putText(src, str("%.3f" % obj[2]), (xmin, ymin), cv.FONT_HERSHEY_SIMPLEX, 0.5, (0, 0, 255), 1, 8)
cv.imshow("Face AR Demo", src)
Landmark检测的方法代码如下:
def infer_landmark(faceImg, landmark_exe_net, input_name, output_name, pw, ph):
# 处理输入图象
rh, rw, rc = faceImg.shape
roi = cv.resize(faceImg, (pw, ph))
roi = roi.transpose(2, 0, 1)
# 推理
prob = landmark_exe_net.infer(inputs={input_name: [roi]})
landmarks = prob[output_name]
pts = np.reshape(landmarks, (-1, 2))
对齐口罩图象与贴图代码如下:
# 寻找点位
for idx, pt in enumerate(pts):
if idx == 18:
left_x = 0 # int(pt[0] * rw)
left_y = int(pt[1] * rh)
# cv.circle(faceImg, (left_x, left_y), 1, (255, 0, 255), 2, 8, 0)
if idx == 34:
right_x = (rw - 1) # int(pt[0] * rw)
right_y = int(pt[1] * rh)
# cv.circle(faceImg, (right_x, right_y), 1, (255, 0, 255), 2, 8, 0)
if idx == 26:
p26_x = int(pt[0] * rw)
p26_y = int(pt[1] * rh)
# 对齐与放缩
dx_t = right_x - left_x
dx_m = anchor_pts[2] - anchor_pts[0]
rate_x = dx_t / dx_m
rate_y = (p26_y - left_y) / (anchor_pts[3] - anchor_pts[1])
dst_mask = cv.resize(mask, (0, 0), fx=rate_x, fy=rate_y)
start_x = np.int(anchor_pts[0] * rate_x);
start_y = np.int(anchor_pts[1] * rate_y);
end_x = np.int(anchor_pts[2] * rate_x)
end_y = np.int(anchor_pts[3] * rate_y)
# 贴图
for row in range(end_y - start_y):
for col in range(end_x - start_x):
b2, g2, r2 = dst_mask[start_y+row, start_x+col]
if b2 < 127 and g2 < 127 and r2 < 127:
faceImg[row+left_y, left_x+col] = ( b2, g2, r2)
最终运行结果如下: