GSoC/2019/StatusReports/ThanhTrungDinh

digiKam AI Face Recognition with OpenCV DNN module

digiKam is KDE desktop application for photos management. For a long time, digiKam team has put a lot of efforts to develop face engine, a feature allowing to scan user photos and suggest face tags automatically basing on pre-tagged faces by users. However, that functionality is currently deactivated in digiKam, as it is slow while not adequately accurate. Thus, this project aims to improve the performance and accuracy of facial recognition in digiKam by exploiting state-of-the-art neural network models in AI and machine learning, combining with highly-optimized OpenCV DNN module.

The project includes 2 main parts:

• Improve face recognition: implementation with OpenCV DNN module
  • reduce processing time while keeping high accuracy
  • classify unknown faces into classes of similar faces
• Improve face detection: implementation to be investigated
  • detect faces across various scales (e.g. big, small, etc.), with occlusion (e.g. sunglasses, scarf, mask etc.), with different orientations (e.g. up, down, left, right, side-face etc.)

Mentors : Maik Qualmann, Gilles Caulier, Stefan Müller

Important Links

Proposal

Project Proposal

Git dev branch

gsoc19-face-recognition

Important commits

• Face Recognition:
  • Label prediction with distance to group of similar faces
  • Particular face alignment to improve recognition accuracy
  • Use cosine similarity to improve recognition accuracy
  • Implementation of face recognition based on OpenFace neural network model with OpenCV DNN
• Face Detection:
  • Face Detection reimplementation with SSD-MobileNet and OpenCV DNN
  • Improvements for SSD-MobileNet
  • Restructure Face Detection module and implement YOLOv3 Face Detection with OpenCV DNN
• Face workflow:
  • Improvements in face workflow and face detection
  • Clustering with k-means
  • Clustering with DBSCAN

Work report

Bonding period (May 6 to May 27)

Generally, I familiarized myself with current Deep Learning (DL) based approach for face recognition in digiKam and picked up the work of Yingjie Liu (digiKam GSoC 2017). His proposal, blog posts and status report led me to FaceNet paper and OpenFace - an opensource implementation of neural network (NN) model inspired by FaceNet paper. They seemed very potential to my work. In addition, Liu also implemented unit tests on his DL implementation and their results (benchmark and accuracy), which can be used as a reference for my work later.

For the rest of the bonding period, I focused on reading carefully FaceNet paper and also looking for other promising NN models to implement when coding period begins. Besides, I also prepared unit tests for face recognition, basing on current test programs and Liu's work.

My plan for next 2 weeks of coding period is to:

• Finish NN model selection
• Finish unit tests
• Start to implement chosen NN model with OpenCV DNN

Coding period : Phase one (May 28 to June 23)

May 28 to June 11 (Week 1 - 2) - Face Recognition got an 8x speed up

I completed my plan for these 2 weeks. I eventually came up with conclusion on using openface pretrained model, as well as the first draft working implementation with OpenCV DNN. On the other hand, I also finished my test codes and benchmarking for face recognition, and tested exhaustively current implementation of face recognition in digiKam, comparing with my new implementation using OpenCV DNN.

Current implementation with Dlib achieves an astonishing accuracy on orl database. It reached above 98% accuracy for 112x92 images in orl database, with only 20% of pre-tagged images. However, it took on average 8s for each image, which is too much. New implementation with OpenCV DNN didn't reach that accuracy, but run much faster. It reached more than 80% of accuracy for 20% of pretagged images. However, it only needed about 1.3 s for each image.

I was able to identify some problems with new implementation as follow:

• Accuracy: It was on prediction phase when euclidean distance was used as a metric to evaluate if a face is closer to another. There are clues that other types of distance (e.g. cosine similarity) which does not require normalized vector may give better results.
• Speed: a file containing model to compute face landmarks is loaded every time a face needed to recognize. In addition, detection is conducted twice on face, one by OpenCV Haar Cascade face detection and other internally by Dlib. All of them are unnecessary and take a lot of time to finish.
• Modularity: different neural network model requires different kind of preprocessing for input images.

More details can be found in my [ blog post].

For final 2 weeks of first coding period, I intends to improve accuracy and speed of that new implementation. On the other hand, I will also investigate the effect of a better face detection model on accuracy of face recognition in digiKam. Scalability problem will be addressed later.

June 11 to June 23 (Week 3 - 4) - Face Recognition achieves above 90% and 100x speed up

For the last 2 weeks, I have well completed my plan:

• By having models loaded only once at the beginning and keeping only Haar Cascade face detection, average time needed is now reduced to around 80ms per image, which is 10x faster than last implementation and 100x faster than current implementation.
• Using cosine similarity instead of euclidean distance has proved an increase in recognition accuracy to 90% for 20% pre-tagged images, which opens a new direction for me.
• I also tested a new approach for face detection with OpenCV DNN and SSD neural network model. The script was on python, but has also proved an accuracy of above 96% for 20% pre-tagged images.

More details can be found in my [ blog post].

Since new implementation of face recognition has achieved some promissing results now, I intend to concentrate on optimization for the next coding phases:

• Modularity: This should be addressed firstly. I will factor and restructure codes so that it will be easier for changing to new neural network models, which require different preprocessing.
• Test and benchmark: I am thinking of changing to new dataset, such as LFW, which can provide better possibility to evaluate the accuracy of new implementation. Therefore, test codes need to be rewritten for better modularity.
• Face detection optimization: Implement neural network approach for face detection with OpenCV DNN.
• Face prediction optimization: Investigate OpenCV FLANN and types of distance indice between vectors. All of that are expected to improve the accuracy for face recognition, as well as introduce the possibility of classify unknown faces into groups of similar faces.

Coding period : Phase two (June 24 to July 21)

For this phase, I have finished some of the work on my check list that I stated at the end of the previous phase. Deciding to optimize to maximum face recognition beforehand, I have concentrated on Modularity and started to implement Face detection optimization. Indeed, changing to new dataset may help to better evaluate the implementation, I thought that it is more for tuning and having statistical results, which should be let to the end of GSoC.

June 24 to July 07 (Week 5 - 6) - Face Recognition is modularized for different neural network models

During these 2 weeks, I restructured the codes, so as to facilitate future development when we need to test different models of neural network for face recognition.

The motivation for this stems from the fact that different models need different preprocessing techniques. Therefore, the codes to load the neural network model with its own preprocessing techniques should be isolated. In addition, this allows the neural network model to be loaded only one time, which improve significantly the performance. Currently, the facesengine only takes from 40 - 60ms to process and recognize a face.

Besides, I delete the old codes using dlib, since after restructuring the codes, there are no places using dlib codes anymore. This is indeed a goal of my GSoC project this year, because this reduces efforts for maintenance, as well as eliminates dlib dependencies, compiler warning and complicated rules for compiler when compiling dlib.

Discussing with my mentors and others digiKam contributors about face recognition, we were all agree that face detection is one of the key factor to improve face recognition in dk. Indeed, current face detection with OpenCV Haar cannot detect face in photos with complicated light condition, shadow or non-frontal faces. When faces cannot be detected, they cannot be recognized. Moreover, the bounding box created by OpenCV Haar are to "small" (i.e. it losts some details on the face). Consequently, this decreases the robust of face recognition.

So for next week, my plan is:

• Study SSD (Single Shot Multibox Detector) with MobileNet model for embedded application
• Implement with OpenCV DNN module

July 08 to July 21 (Week 7 - 8) - Face Detection achieves 100% accuracy on orl test set

For these 2 weeks, I have studied and implemented SSD neural network model.

SSD is one of the neural network models tested with OpenCV DNN models. Therefore, in the github of OpenCV there is a folder supporting face detection with SSD. The model can be downloaded with scripts from this folder. There are also examples on how to implement SSD with OpenCV DNN module.

Implementing SSD, I got a very good result on orl test set, since all faces are detected in comparing with around 90% in case of OpenCV Haar. It also increases the accuracy of face recognition to 96% for 20% of pretagged images. I have not measured the latency for face detection with OpenCV DNN yet, but it seems comparable with OpenCV Haar.

However, when testing on photos downloaded from the internet, the implementation did not work well. Actually, faces are not detected or detected at the wrong places. Therefore, SSD implementation with OpenCV DNN must be studied more.

For the final coding phase of GSoC this year, I intend to work on:

• Face detection optimization: Improve SSD implementation with other photos.
• Face prediction optimization: Investigate OpenCV FLANN and types of distance indice between vectors. In addition, unknown faces must be classified into groups of similar faces.
• Test and benchmark: Changing to LFW dataset. Test codes need to be rewritten for better modularity.

Contacts

Email: [email protected]

Github: TrungDinhT