cs180: proj3

Face Morphing

Part 1: Defining Correspondences

I chose two photos (one of me, one of Betty White!) and took time in Photoshop to crop and align them, ensuring I finalized them to have the same dimensions. To define the corresponding points on the two faces, I used a previous student’s labeling tool (as mentioned in the project spec) to manually select them. I included the four corners of both images so that the triangulation covers the entire image.

From there, I calculated the averages of the corresponding keypoints and used scipy.spatial.Delaunay to create a triangulation of that average shape. Here are the correspondences for both images as well as the average correspondences’ triangulation mesh.

• imgA-me.jpg with correspondences

• imgB-betty.jpg with correspondences

Part 2: Computing the "Mid-Way Face”

Using Part 1’s average shape and triangulation mesh, I iterated over each triangle in the mesh and found the transformation matrix $A$ that transforms the current triangle into the average shape’s corresponding triangle. To compute the inverse warp, I used sk.draw.polygon to extract all pixel coordinates in imgA-me.jpg’s triangles and multiplied them by $A^{-1}$. For values that fall in-between pixel coordinates, I used scipy.interpolate.RegularGridInterpolator (it’s quicker than griddata). As my last step, I took the average of the color of warpedA and warpedB. Below are the original images as well as the midway face.

Note: My mid-way face is pretty creepy for a few reasons

• I used the only photo of mine that has a blank background, but it wasn’t ideal. For example, my head is slightly tilted, my face isn’t expressionless, and I’m wearing glasses.

• I scoured the internet for a celebrity photo that was taken at a similar angle (so that I could at least do some cropping), and this image of Betty White was the best I could find.
  • It does not have a blank background. It has a difficult expression to morph with.

• The transition with our hair isn’t great (I didn’t add any correspondence points on my hair because it worsened the morph, and we were deterred from including non-face points).

• imgA-me.jpg

• imgB-betty.jpg

Part 3: The Morph Sequence

To create my gif (45 frames in 30 fps), I implemented the morph function and changed the warp_frac and dissolve_frac arguments to produce each frame.

Part 4: The "Mean Face" of a Population

Using the IMM Face (Danes) Database, I computed the average shape of the dataset. I decided not to pick a subpopulation since the dataset was so skewed (30 out of 37 were men). From there, I morphed each face in the dataset to the average shape.

• 22-1f.bmp

• 22-1f.bmp morphed into average shape

• 30-1f.bmp

• 30-1f.bmp morphed into average shape

• 35-1f.bmp

• 35-1f.bmp morphed into average shape

• 37-1f.bmp

• 37-1f.bmp morphed into average shape

I computed the average face of the population by taking the average of the morphed faces. Then, I warped an aligned version of my picture (imgA-me-avg.jpg) into the average shape and warped average face into aligned imgA-me-avg.jpg’s geometry.

• imgA-me-avg.jpg

• imgB-avg.jpg

• me warped into average geometry

• average face warped into my geometry

Some funky results! A lot of this is due to the differing camera angles and expressions.

• me-warped-to-average image’s standouts:
  • reduced my wide-open smile since imgB-avg.jpg’s is a more neutral expression with no teeth showing
  • squished nose to match imgB-avg.jpg’s shape
  • more rectangular forehead to match imgB-avg.jpg’s
    • resulted in the weird line on either side of the final image, which is really the pixels from my hairline
  • aligned eyes (mine were slightly tilted so the resulting image corrects that)
    • resulted in a weird warp of my glasses as well

• average-warped-to-me image’s standouts:
  • wider lips to fill in imgA-me-avg.jpg’s smile with teeth
  • warped nose to more closely match mine
  • more curved forehead/hairline
  • slightly tilted, more squinted eyes to match mine (when I smile, my eyes crinkle)

Part 5: Caricatures — Extrapolating from the Mean

Using Part 4’s IMM Face Database average shape, I created caricature-like images of my face by extrapolating from the average shape (with $alpha$ controlling the level of extrapolation). I achieved this by testing values outside of the $[0, 1]$ range.

• $alpha = -0.5$

• $alpha = 1.5$

Definitely can see the difference (exaggerating the mean face’s features versus my own features).

Bells and Whistles: Changing the Gender of My Face

I morphed my face into the mean face of an Indian male for the following 3 scenarios (after rescaling/resizing and manually plotting their correspondence points):

1. Shape: Warp my face into the geometry of the average Indian male’s face.

2. Appearance: Warp average Indian male’s face to my face shape and cross-dissolve.

3. Both: Morph shape and appearance (aka warp + cross-dissolve)

Note: I had to crop my image quite a lot to align with the Indian male mean face.

• overlay points for me

• overlay points for Indian male mean face

• shape only

• appearance only

• shape and appearance

Ran into similar issues discussed in Part 4 (glasses obstruct my facial features like my eyebrows and edges of my face, my slightly tilted eyes also impact the quality of the warp, differing camera angles since I used the same photo throughout this project, etc).

Reflection & Bloopers

Fun (and sometimes slightly creepy lol) project where I learned a lot! The results are definitely dependent on which photos you choose initially, and mine were definitely quite different. I found there’s only so much you can do with resizing in Photoshop, and my struggle came with camera angles (and my slight head tilt). Here are some bloopers I saved from this project (specifically from Part 4).

• initial workaround produced this when I ran into blending issues (for pixels on the triangulation mesh’s lines)!

• flipped my coordinates when trying to produce the average image for the population