Lip Colour Finder – Profiling for Speedup

I’ve been working to improve the turnaround time for Lip Colour Finder results. In order to find the areas of my code most amenable to tweaking I turned to profiling.

A few months ago I came across a profiler for Python on Hacker News and decided to tuck it away for future projects. The name of the profiler is Py-Spy and it provides a real-time top-like view of the innards your program. Py-Spy can also visualize where your code is spending its time through flame graphs (Py-Spy produces the icicle graph variant).

Flame graphs show the stack profile population on the x-axis and stack depth on the y-axis. The wider a square on the flame graph, the more time the code it represents spent on the stack. Below is the flame graph generated by Py-Spy while using Selenium to stress test the site (I will detail this in a future blog post):

Near the bottom right of the graph you can see the most relevant information. Upon a POST request my Django view calls find_lip_color.py (pardon the inconsistency between color and colour :D). Within that module is a function called detect_skin. From the graph you can see that that function spends quite a bit of time on the stack. Using this information I decided to start here for optimization.

The detect_skin function employs the skin detection method defined here and is used to generate an estimate of a face’s skin colour. I timed each component of the function when analyzing the following images using Python’s perf_counter in order to narrow down the worst offender (Spoiler: you can probably tell from the flame graph):

Test Image 1 [1]

Test Image 2 [2]

Here is the timing for Test Image 1:

detect_edges_skimage: 0.308 seconds
gray2bw_skimage: 0.001 seconds
find_facial_features: 0.001 seconds
create_face_ellipse_skimage: 0.002 seconds
mask_edges_skimage: 0.002 seconds
rgb2hsv: 0.055 seconds
hsv_image crop: 0.002 seconds
generate_hsv_2d_histogram: 0.003 seconds
generate_bounds: 0.00006 seconds
full analysis: 0.5156 seconds

And for Test Image 2:

detect_edges_skimage: 0.694 seconds
gray2bw_skimage: 0.002 seconds
find_facial_features: 0.002 seconds
create_face_ellipse_skimage: 0.002 seconds
mask_edges_skimage: 0.008 seconds
rgb2hsv: 0.136 seconds
hsv_image crop: 0.002 seconds
generate_hsv_2d_histogram: 0.006 seconds
generate_bounds: 0.00005 seconds
full analysis: 2.9326 seconds

The full analysis line gives the total time required to return a lip colour estimate. That line is quite different for each image due to differences in their resolutions. The key take-home is that the detect_edges_skimage component of detect_skin requires a huge portion of the full analysis time. 23.7% for Test Image 2 and 59.7% for Test Image 1. Improving the speed of this component will go a long way for improving result turnaround times.

The detect_edges_skimage component finds edges in the supplied images and processes the edge output to create contiguous edge lines. High quality edge detection is crucial for accurate colour estimates as it is a vital part of the method I use for determining the skin colour of a supplied face. In order to ensure that colour estimation quality is maintained I created a test infrastructure that reports differences in colour estimates compared to a baseline for a suite of test images (I will detail this infrastructure in a future blog post).

Due to the importance and complexity of the edge detection component I have held off on optimizing it for now. However while looking through the detect_skin function I found that I was converting the entire supplied image to the HSV colour space when only a crop of the image needed the conversion (the HSV colour space allows for easier skin tone segmentation). By tweaking the relevant portions of detect_skin I was able to produce the following times:

Test Image 1:

detect_edges_skimage: 0.309 seconds
gray2bw_skimage: 0.001 seconds
find_facial_features: 0.001 seconds
create_face_ellipse_skimage: 0.002 seconds
mask_edges_skimage: 0.002 seconds
rgb2hsv: 0.016 seconds
hsv_image reshape: 0.00001 seconds
generate_hsv_2d_histogram: 0.003 seconds
generate_bounds: 0.00006 seconds
full analysis: 0.4768 seconds

Test Image 2:

detect_edges_skimage: 0.685 seconds
gray2bw_skimage: 0.002 seconds
find_facial_features: 0.002 seconds
create_face_ellipse_skimage: 0.002 seconds
mask_edges_skimage: 0.008 seconds
rgb2hsv: 0.032 seconds
hsv_image reshape: 0.00001 seconds
generate_hsv_2d_histogram: 0.005 seconds
generate_bounds: 0.00005 seconds
full analysis: 2.7896 seconds

When multiple runs were averaged the reductions in overall estimate turnaround time were 7.8% and 4.8% for Test Image 1 and Test Image 2 respectively (with no impact to colour estimates according to my test infrastructure).

Profiling allowed me to find areas of the code where performance is being left on the table. I will be applying it in the coming weeks to improve the efficiency of my code (particularly in edge detection) and reduce overall turnaround time for lipstick recommendations.