eos  0.7.1
Classes | Enumerations | Functions
eos::fitting Namespace Reference

Pose and shape fitting of a 3D Morphable Model. More...

Classes

struct  ContourLandmarks
 Defines which 2D landmarks comprise the right and left face contour. More...
 
struct  Frustum
 A class representing a camera viewing frustum. At the moment used as orthographic camera only. More...
 
struct  ModelContour
 Definition of the vertex indices that define the right and left model contour. More...
 
struct  RenderingParameters
 Represents a set of estimated model parameters (rotation, translation) and camera parameters (viewing frustum). More...
 

Enumerations

enum  CameraType { Orthographic, Perspective }
 Type of a camera (projection). More...
 

Functions

cv::Mat estimate_affine_camera (std::vector< cv::Vec2f > image_points, std::vector< cv::Vec4f > model_points)
 
cv::Vec2f project_affine (cv::Vec4f vertex, cv::Mat affine_camera_matrix, int screen_width, int screen_height)
 
std::vector< float > fit_blendshapes_to_landmarks_linear (std::vector< eos::morphablemodel::Blendshape > blendshapes, cv::Mat face_instance, cv::Mat affine_camera_matrix, std::vector< cv::Vec2f > landmarks, std::vector< int > vertex_ids, float lambda=500.0f)
 
std::pair< std::vector< std::string >, std::vector< int > > select_contour (float yaw_angle, const ContourLandmarks &contour_landmarks, const ModelContour &model_contour)
 
std::tuple< std::vector< cv::Vec2f >, std::vector< cv::Vec4f >, std::vector< int > > get_nearest_contour_correspondences (const eos::core::LandmarkCollection< cv::Vec2f > &landmarks, const std::vector< std::string > &landmark_contour_identifiers, const std::vector< int > &model_contour_indices, const morphablemodel::MorphableModel &morphable_model, const glm::mat4x4 &view_model, const glm::mat4x4 &ortho_projection, const glm::vec4 &viewport)
 
std::tuple< std::vector< cv::Vec2f >, std::vector< cv::Vec4f >, std::vector< int > > get_contour_correspondences (const eos::core::LandmarkCollection< cv::Vec2f > &landmarks, const ContourLandmarks &contour_landmarks, const ModelContour &model_contour, float yaw_angle, const morphablemodel::MorphableModel &morphable_model, const glm::mat4x4 &view_model, const glm::mat4x4 &ortho_projection, const glm::vec4 &viewport)
 
cv::Mat fit_shape (cv::Mat affine_camera_matrix, eos::morphablemodel::MorphableModel morphable_model, std::vector< eos::morphablemodel::Blendshape > blendshapes, std::vector< cv::Vec2f > image_points, std::vector< int > vertex_indices, float lambda, boost::optional< int > num_coefficients_to_fit, std::vector< float > &pca_shape_coefficients, std::vector< float > &blendshape_coefficients)
 
cv::Mat fit_shape (cv::Mat affine_camera_matrix, eos::morphablemodel::MorphableModel morphable_model, std::vector< eos::morphablemodel::Blendshape > blendshapes, std::vector< cv::Vec2f > image_points, std::vector< int > vertex_indices, float lambda=3.0f, boost::optional< int > num_coefficients_to_fit=boost::optional< int >())
 
std::vector< float > fit_shape_to_landmarks_linear (morphablemodel::MorphableModel morphable_model, cv::Mat affine_camera_matrix, std::vector< cv::Vec2f > landmarks, std::vector< int > vertex_ids, cv::Mat base_face=cv::Mat(), float lambda=3.0f, boost::optional< int > num_coefficients_to_fit=boost::optional< int >(), boost::optional< float > detector_standard_deviation=boost::optional< float >(), boost::optional< float > model_standard_deviation=boost::optional< float >())
 
void save_rendering_parameters (RenderingParameters rendering_parameters, std::string filename)
 
cv::Mat to_mat (const glm::mat4x4 &glm_matrix)
 Converts a glm::mat4x4 to a cv::Mat. More...
 
glm::mat4x4 get_4x4_modelview_matrix (fitting::RenderingParameters params)
 Creates a 4x4 model-view matrix from given fitting parameters. More...
 
cv::Mat get_3x4_affine_camera_matrix (fitting::RenderingParameters params, int width, int height)
 Creates a 3x4 affine camera matrix from given fitting parameters. The matrix transforms points directly from model-space to screen-space. More...
 
glm::vec4 get_opencv_viewport (int width, int height)
 Returns a glm/OpenGL compatible viewport vector that flips y and has the origin on the top-left, like in OpenCV. More...
 
RenderingParameters estimate_orthographic_camera (std::vector< cv::Vec2f > image_points, std::vector< cv::Vec4f > model_points, int width, int height)
 This algorithm estimates the rotation angles and translation of the model, as well as the viewing frustum of the camera, given a set of corresponding 2D-3D points. More...
 

Detailed Description

Pose and shape fitting of a 3D Morphable Model.

Enumeration Type Documentation

Type of a camera (projection).

Currently either orthographic or perspective. Used in RenderingParameters.

Function Documentation

cv::Mat eos::fitting::estimate_affine_camera ( std::vector< cv::Vec2f >  image_points,
std::vector< cv::Vec4f >  model_points 
)

The Gold Standard Algorithm for estimating an affine camera matrix from world to image correspondences. See Algorithm 7.2 in Multiple View Geometry, Hartley & Zisserman, 2nd Edition, 2003.

Requires >= 4 corresponding points.

The estimated camera matrix works together with render::render_affine(Mesh, cv::Mat, int, int, bool) to for example render the model or extract texture from the image.

Parameters
[in]image_pointsA list of 2D image points.
[in]model_pointsCorresponding points of a 3D model.
Returns
A 3x4 affine camera matrix (the third row is [0, 0, 0, 1]).
RenderingParameters eos::fitting::estimate_orthographic_camera ( std::vector< cv::Vec2f >  image_points,
std::vector< cv::Vec4f >  model_points,
int  width,
int  height 
)

This algorithm estimates the rotation angles and translation of the model, as well as the viewing frustum of the camera, given a set of corresponding 2D-3D points.

It assumes an orthographic camera and estimates 6 parameters, [r_x, r_y, r_z, t_x, t_y, frustum_scale], where the first five describe how to transform the model, and the last one describes the cameras viewing frustum (see CameraParameters). This 2D-3D correspondence problem is solved using Eigen's LevenbergMarquardt algorithm.

The method is slightly inspired by "Computer Vision: Models Learning and Inference", Simon J.D. Prince, 2012, but different in a lot of respects.

Eigen's LM implementation requires at least 6 data points, so we require >= 6 corresponding points.

Notes/improvements: The algorithm works reliable as it is, however, it could be improved with the following:

  • A better initial guess (see e.g. Prince)
  • Using the analytic derivatives instead of Eigen::NumericalDiff - they're easy to calculate.
Parameters
[in]image_pointsA list of 2D image points.
[in]model_pointsCorresponding points of a 3D model.
[in]widthWidth of the image (or viewport).
[in]heightHeight of the image (or viewport).
Returns
The estimated model and camera parameters.
std::vector<float> eos::fitting::fit_blendshapes_to_landmarks_linear ( std::vector< eos::morphablemodel::Blendshape blendshapes,
cv::Mat  face_instance,
cv::Mat  affine_camera_matrix,
std::vector< cv::Vec2f >  landmarks,
std::vector< int >  vertex_ids,
float  lambda = 500.0f 
)
inline

Fits blendshape coefficients to given 2D landmarks, given a current face shape instance. It's a linear, closed-form solution fitting algorithm, with regularisation (constraining the L2-norm of the coefficients).

This algorithm is very similar to the shape fitting in fit_shape_to_landmarks_linear. Instead of the PCA basis, the blendshapes are used, and instead of the mean, a current face instance is used to do the fitting from.

Parameters
[in]blendshapesA vector with blendshapes to estimate the coefficients for.
[in]face_instanceA shape instance from which the blendshape coefficients should be estimated (i.e. the current mesh without expressions, e.g. estimated from a previous PCA-model fitting). A 3m x 1 matrix.
[in]affine_camera_matrixA 3x4 affine camera matrix from model to screen-space (should probably be of type CV_32FC1 as all our calculations are done with float).
[in]landmarks2D landmarks from an image to fit the blendshapes to.
[in]vertex_idsThe vertex ids in the model that correspond to the 2D points.
[in]lambdaA regularisation parameter, constraining the L2-norm of the coefficients.
Returns
The estimated blendshape-coefficients.
cv::Mat eos::fitting::fit_shape ( cv::Mat  affine_camera_matrix,
eos::morphablemodel::MorphableModel  morphable_model,
std::vector< eos::morphablemodel::Blendshape blendshapes,
std::vector< cv::Vec2f >  image_points,
std::vector< int >  vertex_indices,
float  lambda,
boost::optional< int >  num_coefficients_to_fit,
std::vector< float > &  pca_shape_coefficients,
std::vector< float > &  blendshape_coefficients 
)

Convenience function that fits the shape model and expression blendshapes to landmarks. Makes the fitted PCA shape and blendshape coefficients accessible via the out parameters pca_shape_coefficients and blendshape_coefficients. It iterates PCA-shape and blendshape fitting until convergence (usually it converges within 5 to 10 iterations).

See fit_shape_model(cv::Mat, eos::morphablemodel::MorphableModel, std::vector<eos::morphablemodel::Blendshape>, std::vector<cv::Vec2f>, std::vector<int>, float lambda) for a simpler overload that just returns the shape instance.

Parameters
[in]affine_camera_matrixThe estimated pose as a 3x4 affine camera matrix that is used to fit the shape.
[in]morphable_modelThe 3D Morphable Model used for the shape fitting.
[in]blendshapesA vector of blendshapes that are being fit to the landmarks in addition to the PCA model.
[in]image_points2D landmarks from an image to fit the model to.
[in]vertex_indicesThe vertex indices in the model that correspond to the 2D points.
[in]lambdaRegularisation parameter of the PCA shape fitting.
[in]num_coefficients_to_fitHow many shape-coefficients to fit (all others will stay 0). Should be bigger than zero, or boost::none to fit all coefficients.
[out]pca_shape_coefficientsOutput parameter that will contain the resulting pca shape coefficients.
[out]blendshape_coefficientsOutput parameter that will contain the resulting blendshape coefficients.
Returns
The fitted model shape instance.
cv::Mat eos::fitting::fit_shape ( cv::Mat  affine_camera_matrix,
eos::morphablemodel::MorphableModel  morphable_model,
std::vector< eos::morphablemodel::Blendshape blendshapes,
std::vector< cv::Vec2f >  image_points,
std::vector< int >  vertex_indices,
float  lambda = 3.0f,
boost::optional< int >  num_coefficients_to_fit = boost::optional<int>() 
)

Convenience function that fits the shape model and expression blendshapes to landmarks. It iterates PCA-shape and blendshape fitting until convergence (usually it converges within 5 to 10 iterations).

Parameters
[in]affine_camera_matrixThe estimated pose as a 3x4 affine camera matrix that is used to fit the shape.
[in]morphable_modelThe 3D Morphable Model used for the shape fitting.
[in]blendshapesA vector of blendshapes that are being fit to the landmarks in addition to the PCA model.
[in]image_points2D landmarks from an image to fit the model to.
[in]vertex_indicesThe vertex indices in the model that correspond to the 2D points.
[in]lambdaRegularisation parameter of the PCA shape fitting.
[in]num_coefficients_to_fitHow many shape-coefficients to fit (all others will stay 0). Should be bigger than zero, or boost::none to fit all coefficients.
Returns
The fitted model shape instance.
std::vector<float> eos::fitting::fit_shape_to_landmarks_linear ( morphablemodel::MorphableModel  morphable_model,
cv::Mat  affine_camera_matrix,
std::vector< cv::Vec2f >  landmarks,
std::vector< int >  vertex_ids,
cv::Mat  base_face = cv::Mat(),
float  lambda = 3.0f,
boost::optional< int >  num_coefficients_to_fit = boost::optional<int>(),
boost::optional< float >  detector_standard_deviation = boost::optional<float>(),
boost::optional< float >  model_standard_deviation = boost::optional<float>() 
)
inline

Fits the shape of a Morphable Model to given 2D landmarks (i.e. estimates the maximum likelihood solution of the shape coefficients) as proposed in [1]. It's a linear, closed-form solution fitting of the shape, with regularisation (prior towards the mean).

[1] O. Aldrian & W. Smith, Inverse Rendering of Faces with a 3D Morphable Model, PAMI 2013.

Note: Using less than the maximum number of coefficients to fit is not thoroughly tested yet and may contain an error. Note: Returns coefficients following standard normal distribution (i.e. all have similar magnitude). Why? Because we fit using the normalised basis? Note: The standard deviations given should be a vector, i.e. different for each landmark. This is not implemented yet.

Parameters
[in]morphable_modelThe Morphable Model whose shape (coefficients) are estimated.
[in]affine_camera_matrixA 3x4 affine camera matrix from model to screen-space (should probably be of type CV_32FC1 as all our calculations are done with float).
[in]landmarks2D landmarks from an image to fit the model to.
[in]vertex_idsThe vertex ids in the model that correspond to the 2D points.
[in]base_faceThe base or reference face from where the fitting is started. Usually this would be the models mean face, which is what will be used if the parameter is not explicitly specified.
[in]lambdaThe regularisation parameter (weight of the prior towards the mean).
[in]num_coefficients_to_fitHow many shape-coefficients to fit (all others will stay 0). Should be bigger than zero, or boost::none to fit all coefficients.
[in]detector_standard_deviationThe standard deviation of the 2D landmarks given (e.g. of the detector used), in pixels.
[in]model_standard_deviationThe standard deviation of the 3D vertex points in the 3D model, projected to 2D (so the value is in pixels).
Returns
The estimated shape-coefficients (alphas).
cv::Mat eos::fitting::get_3x4_affine_camera_matrix ( fitting::RenderingParameters  params,
int  width,
int  height 
)

Creates a 3x4 affine camera matrix from given fitting parameters. The matrix transforms points directly from model-space to screen-space.

This function is mainly used since the linear shape fitting fitting::fit_shape_to_landmarks_linear expects one of these 3x4 affine camera matrices, as well as render::extract_texture.

glm::mat4x4 eos::fitting::get_4x4_modelview_matrix ( fitting::RenderingParameters  params)

Creates a 4x4 model-view matrix from given fitting parameters.

Together with the Frustum information, this describes the full orthographic rendering parameters of the OpenGL pipeline. Example:

fitting::OrthographicRenderingParameters rendering_params = ...;
glm::mat4x4 view_model = get_4x4_modelview_matrix(rendering_params);
glm::mat4x4 ortho_projection = glm::ortho(rendering_params.frustum.l, rendering_params.frustum.r, rendering_params.frustum.b, rendering_params.frustum.t);
glm::vec4 viewport(0, image.rows, image.cols, -image.rows); // flips y, origin top-left, like in OpenCV
// project a point from 3D to 2D:
glm::vec3 point_3d = ...; // from a mesh for example
glm::vec3 point_2d = glm::project(point_3d, view_model, ortho_projection, viewport);
std::tuple<std::vector<cv::Vec2f>, std::vector<cv::Vec4f>, std::vector<int> > eos::fitting::get_contour_correspondences ( const eos::core::LandmarkCollection< cv::Vec2f > &  landmarks,
const ContourLandmarks contour_landmarks,
const ModelContour model_contour,
float  yaw_angle,
const morphablemodel::MorphableModel morphable_model,
const glm::mat4x4 &  view_model,
const glm::mat4x4 &  ortho_projection,
const glm::vec4 &  viewport 
)

Given a set of 2D image landmarks, finds the closest (in a L2 sense) 3D vertex from a list of vertices pre-defined in model_contour. landmarks can contain all landmarks, and the function will sub-select the relevant contour landmarks with the help of the given contour_landmarks. This function choses the front-facing contour and only fits this contour to the 3D model, since these correspondences are approximately static and do not move with changing pose-angle.

It's the main contour fitting function that calls all other functions.

Note: Maybe rename to find_contour_correspondences, to highlight that there is (potentially a lot) computational cost involved? Note: Does ortho_projection have to be specifically orthographic? Otherwise, if it works with perspective too, rename to just "projection".

Parameters
[in]landmarksAll image landmarks.
[in]contour_landmarks2D image contour ids of left or right side (for example for ibug landmarks).
[in]model_contourThe model contour indices that should be considered to find the closest corresponding 3D vertex.
[in]yaw_angleYaw angle of the current fitting. The front-facing contour will be chosen depending on this yaw angle.
[in]morphable_modelA Morphable Model whose mean is used.
[in]view_modelModel-view matrix of the current fitting to project the 3D model vertices to 2D.
[in]ortho_projectionProjection matrix to project the 3D model vertices to 2D.
[in]viewportCurrent viewport to use.
Returns
A tuple with the 2D contour landmark points, the corresponding points in the 3D shape model and their vertex indices.
std::tuple< std::vector< cv::Vec2f >, std::vector< cv::Vec4f >, std::vector< int > > eos::fitting::get_nearest_contour_correspondences ( const eos::core::LandmarkCollection< cv::Vec2f > &  landmarks,
const std::vector< std::string > &  landmark_contour_identifiers,
const std::vector< int > &  model_contour_indices,
const morphablemodel::MorphableModel morphable_model,
const glm::mat4x4 &  view_model,
const glm::mat4x4 &  ortho_projection,
const glm::vec4 &  viewport 
)

Given a set of 2D image landmarks, finds the closest (in a L2 sense) 3D vertex from a list of vertices pre-defined in model_contour. Assumes to be given contour correspondences of the front-facing contour.

Note: Maybe rename to find_nearest_contour_points, to highlight that there is (potentially a lot) computational cost involved? Note: Does ortho_projection have to be specifically orthographic? Otherwise, if it works with perspective too, rename to just "projection". More notes: Actually, only return the vertex id, not the point? Same with get_corresponding_pointset? Because then it's much easier to use the current shape estimate instead of the mean! But this function needs to project. So... it should take a Mesh actually? But creating a Mesh is a lot of computation? When we want to use the non-mean, then we need to use draw_sample() anyway? So overhead of Mesh is only if we use the mean? Maybe two overloads? Note: Uses the mean to calculate.

Parameters
[in]landmarksAll image landmarks.
[in]landmark_contour_identifiers2D image contour ids of left or right side (for example for ibug landmarks).
[in]model_contour_indicesThe model contour indices that should be considered to find the closest corresponding 3D vertex.
[in]morphable_modelThe Morphable Model whose shape (coefficients) are estimated.
[in]view_modelModel-view matrix of the current fitting to project the 3D model vertices to 2D.
[in]ortho_projectionProjection matrix to project the 3D model vertices to 2D.
[in]viewportCurrent viewport to use.
Returns
A tuple with the 2D contour landmark points, the corresponding points in the 3D shape model and their vertex indices.
glm::vec4 eos::fitting::get_opencv_viewport ( int  width,
int  height 
)

Returns a glm/OpenGL compatible viewport vector that flips y and has the origin on the top-left, like in OpenCV.

Note: Move to detail namespace / not used at the moment.

cv::Vec2f eos::fitting::project_affine ( cv::Vec4f  vertex,
cv::Mat  affine_camera_matrix,
int  screen_width,
int  screen_height 
)
inline

Projects a point from world coordinates to screen coordinates. First, an estimated affine camera matrix is used to transform the point to clip space. Second, the point is transformed to screen coordinates using the window transform. The window transform also flips the y-axis (the image origin is top-left, while in clip space top is +1 and bottom is -1).

Note: Assumes the affine camera matrix only projects from world to clip space, because a subsequent window transform is applied. Todo: This is outdated, now that we estimate the matrix from world to screen space directly.

Parameters
[in]vertexA vertex in 3D space. vertex[3] = 1.0f.
[in]affine_camera_matrixA 3x4 affine camera matrix.
[in]screen_widthWidth of the screen or window used for projection.
[in]screen_heightHeight of the screen or window used for projection.
Returns
A vector with x and y coordinates transformed to screen coordinates.
void eos::fitting::save_rendering_parameters ( RenderingParameters  rendering_parameters,
std::string  filename 
)

Saves the rendering parameters for an image to a json file.

Parameters
[in]rendering_parametersAn instance of class RenderingParameters.
[in]filenameThe file to write.
Exceptions
std::runtime_errorif unable to open the given file for writing.
std::pair< std::vector< std::string >, std::vector< int > > eos::fitting::select_contour ( float  yaw_angle,
const ContourLandmarks contour_landmarks,
const ModelContour model_contour 
)

Takes a set of 2D and 3D contour landmarks and a yaw angle and returns two vectors with either the right or the left 2D and 3D contour indices. This function does not establish correspondence between the 2D and 3D landmarks, it just selects the front-facing contour. The two returned vectors can thus have different size. Correspondence can be established using get_nearest_contour_correspondences().

Note: Maybe rename to find_nearest_contour_points, to highlight that there is (potentially a lot) computational cost involved?

Parameters
[in]yaw_angleyaw angle in degrees.
[in]contour_landmarks2D image contour ids of left or right side (for example for ibug landmarks).
[in]model_contourThe model contour indices that should be used/considered to find the closest corresponding 3D vertex.
Returns
A pair with two vectors containing the selected 2D image contour landmark ids and the 3D model contour indices.
cv::Mat eos::fitting::to_mat ( const glm::mat4x4 &  glm_matrix)

Converts a glm::mat4x4 to a cv::Mat.

Note: move to render namespace