eos
0.7.1
|
Pose and shape fitting of a 3D Morphable Model. More...
Classes | |
struct | ContourLandmarks |
Defines which 2D landmarks comprise the right and left face contour. More... | |
struct | Frustum |
A class representing a camera viewing frustum. At the moment used as orthographic camera only. More... | |
struct | ModelContour |
Definition of the vertex indices that define the right and left model contour. More... | |
struct | RenderingParameters |
Represents a set of estimated model parameters (rotation, translation) and camera parameters (viewing frustum). More... | |
Enumerations | |
enum | CameraType { Orthographic, Perspective } |
Type of a camera (projection). More... | |
Functions | |
cv::Mat | estimate_affine_camera (std::vector< cv::Vec2f > image_points, std::vector< cv::Vec4f > model_points) |
cv::Vec2f | project_affine (cv::Vec4f vertex, cv::Mat affine_camera_matrix, int screen_width, int screen_height) |
std::vector< float > | fit_blendshapes_to_landmarks_linear (std::vector< eos::morphablemodel::Blendshape > blendshapes, cv::Mat face_instance, cv::Mat affine_camera_matrix, std::vector< cv::Vec2f > landmarks, std::vector< int > vertex_ids, float lambda=500.0f) |
std::pair< std::vector< std::string >, std::vector< int > > | select_contour (float yaw_angle, const ContourLandmarks &contour_landmarks, const ModelContour &model_contour) |
std::tuple< std::vector< cv::Vec2f >, std::vector< cv::Vec4f >, std::vector< int > > | get_nearest_contour_correspondences (const eos::core::LandmarkCollection< cv::Vec2f > &landmarks, const std::vector< std::string > &landmark_contour_identifiers, const std::vector< int > &model_contour_indices, const morphablemodel::MorphableModel &morphable_model, const glm::mat4x4 &view_model, const glm::mat4x4 &ortho_projection, const glm::vec4 &viewport) |
std::tuple< std::vector< cv::Vec2f >, std::vector< cv::Vec4f >, std::vector< int > > | get_contour_correspondences (const eos::core::LandmarkCollection< cv::Vec2f > &landmarks, const ContourLandmarks &contour_landmarks, const ModelContour &model_contour, float yaw_angle, const morphablemodel::MorphableModel &morphable_model, const glm::mat4x4 &view_model, const glm::mat4x4 &ortho_projection, const glm::vec4 &viewport) |
cv::Mat | fit_shape (cv::Mat affine_camera_matrix, eos::morphablemodel::MorphableModel morphable_model, std::vector< eos::morphablemodel::Blendshape > blendshapes, std::vector< cv::Vec2f > image_points, std::vector< int > vertex_indices, float lambda, boost::optional< int > num_coefficients_to_fit, std::vector< float > &pca_shape_coefficients, std::vector< float > &blendshape_coefficients) |
cv::Mat | fit_shape (cv::Mat affine_camera_matrix, eos::morphablemodel::MorphableModel morphable_model, std::vector< eos::morphablemodel::Blendshape > blendshapes, std::vector< cv::Vec2f > image_points, std::vector< int > vertex_indices, float lambda=3.0f, boost::optional< int > num_coefficients_to_fit=boost::optional< int >()) |
std::vector< float > | fit_shape_to_landmarks_linear (morphablemodel::MorphableModel morphable_model, cv::Mat affine_camera_matrix, std::vector< cv::Vec2f > landmarks, std::vector< int > vertex_ids, cv::Mat base_face=cv::Mat(), float lambda=3.0f, boost::optional< int > num_coefficients_to_fit=boost::optional< int >(), boost::optional< float > detector_standard_deviation=boost::optional< float >(), boost::optional< float > model_standard_deviation=boost::optional< float >()) |
void | save_rendering_parameters (RenderingParameters rendering_parameters, std::string filename) |
cv::Mat | to_mat (const glm::mat4x4 &glm_matrix) |
Converts a glm::mat4x4 to a cv::Mat. More... | |
glm::mat4x4 | get_4x4_modelview_matrix (fitting::RenderingParameters params) |
Creates a 4x4 model-view matrix from given fitting parameters. More... | |
cv::Mat | get_3x4_affine_camera_matrix (fitting::RenderingParameters params, int width, int height) |
Creates a 3x4 affine camera matrix from given fitting parameters. The matrix transforms points directly from model-space to screen-space. More... | |
glm::vec4 | get_opencv_viewport (int width, int height) |
Returns a glm/OpenGL compatible viewport vector that flips y and has the origin on the top-left, like in OpenCV. More... | |
RenderingParameters | estimate_orthographic_camera (std::vector< cv::Vec2f > image_points, std::vector< cv::Vec4f > model_points, int width, int height) |
This algorithm estimates the rotation angles and translation of the model, as well as the viewing frustum of the camera, given a set of corresponding 2D-3D points. More... | |
Pose and shape fitting of a 3D Morphable Model.
|
strong |
Type of a camera (projection).
Currently either orthographic or perspective. Used in RenderingParameters.
cv::Mat eos::fitting::estimate_affine_camera | ( | std::vector< cv::Vec2f > | image_points, |
std::vector< cv::Vec4f > | model_points | ||
) |
The Gold Standard Algorithm for estimating an affine camera matrix from world to image correspondences. See Algorithm 7.2 in Multiple View Geometry, Hartley & Zisserman, 2nd Edition, 2003.
Requires >= 4 corresponding points.
The estimated camera matrix works together with render::render_affine(Mesh, cv::Mat, int, int, bool) to for example render the model or extract texture from the image.
[in] | image_points | A list of 2D image points. |
[in] | model_points | Corresponding points of a 3D model. |
RenderingParameters eos::fitting::estimate_orthographic_camera | ( | std::vector< cv::Vec2f > | image_points, |
std::vector< cv::Vec4f > | model_points, | ||
int | width, | ||
int | height | ||
) |
This algorithm estimates the rotation angles and translation of the model, as well as the viewing frustum of the camera, given a set of corresponding 2D-3D points.
It assumes an orthographic camera and estimates 6 parameters, [r_x, r_y, r_z, t_x, t_y, frustum_scale], where the first five describe how to transform the model, and the last one describes the cameras viewing frustum (see CameraParameters). This 2D-3D correspondence problem is solved using Eigen's LevenbergMarquardt algorithm.
The method is slightly inspired by "Computer Vision: Models Learning and Inference", Simon J.D. Prince, 2012, but different in a lot of respects.
Eigen's LM implementation requires at least 6 data points, so we require >= 6 corresponding points.
Notes/improvements: The algorithm works reliable as it is, however, it could be improved with the following:
[in] | image_points | A list of 2D image points. |
[in] | model_points | Corresponding points of a 3D model. |
[in] | width | Width of the image (or viewport). |
[in] | height | Height of the image (or viewport). |
|
inline |
Fits blendshape coefficients to given 2D landmarks, given a current face shape instance. It's a linear, closed-form solution fitting algorithm, with regularisation (constraining the L2-norm of the coefficients).
This algorithm is very similar to the shape fitting in fit_shape_to_landmarks_linear. Instead of the PCA basis, the blendshapes are used, and instead of the mean, a current face instance is used to do the fitting from.
[in] | blendshapes | A vector with blendshapes to estimate the coefficients for. |
[in] | face_instance | A shape instance from which the blendshape coefficients should be estimated (i.e. the current mesh without expressions, e.g. estimated from a previous PCA-model fitting). A 3m x 1 matrix. |
[in] | affine_camera_matrix | A 3x4 affine camera matrix from model to screen-space (should probably be of type CV_32FC1 as all our calculations are done with float). |
[in] | landmarks | 2D landmarks from an image to fit the blendshapes to. |
[in] | vertex_ids | The vertex ids in the model that correspond to the 2D points. |
[in] | lambda | A regularisation parameter, constraining the L2-norm of the coefficients. |
cv::Mat eos::fitting::fit_shape | ( | cv::Mat | affine_camera_matrix, |
eos::morphablemodel::MorphableModel | morphable_model, | ||
std::vector< eos::morphablemodel::Blendshape > | blendshapes, | ||
std::vector< cv::Vec2f > | image_points, | ||
std::vector< int > | vertex_indices, | ||
float | lambda, | ||
boost::optional< int > | num_coefficients_to_fit, | ||
std::vector< float > & | pca_shape_coefficients, | ||
std::vector< float > & | blendshape_coefficients | ||
) |
Convenience function that fits the shape model and expression blendshapes to landmarks. Makes the fitted PCA shape and blendshape coefficients accessible via the out parameters pca_shape_coefficients
and blendshape_coefficients
. It iterates PCA-shape and blendshape fitting until convergence (usually it converges within 5 to 10 iterations).
See fit_shape_model(cv::Mat, eos::morphablemodel::MorphableModel, std::vector<eos::morphablemodel::Blendshape>, std::vector<cv::Vec2f>, std::vector<int>, float lambda) for a simpler overload that just returns the shape instance.
[in] | affine_camera_matrix | The estimated pose as a 3x4 affine camera matrix that is used to fit the shape. |
[in] | morphable_model | The 3D Morphable Model used for the shape fitting. |
[in] | blendshapes | A vector of blendshapes that are being fit to the landmarks in addition to the PCA model. |
[in] | image_points | 2D landmarks from an image to fit the model to. |
[in] | vertex_indices | The vertex indices in the model that correspond to the 2D points. |
[in] | lambda | Regularisation parameter of the PCA shape fitting. |
[in] | num_coefficients_to_fit | How many shape-coefficients to fit (all others will stay 0). Should be bigger than zero, or boost::none to fit all coefficients. |
[out] | pca_shape_coefficients | Output parameter that will contain the resulting pca shape coefficients. |
[out] | blendshape_coefficients | Output parameter that will contain the resulting blendshape coefficients. |
cv::Mat eos::fitting::fit_shape | ( | cv::Mat | affine_camera_matrix, |
eos::morphablemodel::MorphableModel | morphable_model, | ||
std::vector< eos::morphablemodel::Blendshape > | blendshapes, | ||
std::vector< cv::Vec2f > | image_points, | ||
std::vector< int > | vertex_indices, | ||
float | lambda = 3.0f , |
||
boost::optional< int > | num_coefficients_to_fit = boost::optional<int>() |
||
) |
Convenience function that fits the shape model and expression blendshapes to landmarks. It iterates PCA-shape and blendshape fitting until convergence (usually it converges within 5 to 10 iterations).
[in] | affine_camera_matrix | The estimated pose as a 3x4 affine camera matrix that is used to fit the shape. |
[in] | morphable_model | The 3D Morphable Model used for the shape fitting. |
[in] | blendshapes | A vector of blendshapes that are being fit to the landmarks in addition to the PCA model. |
[in] | image_points | 2D landmarks from an image to fit the model to. |
[in] | vertex_indices | The vertex indices in the model that correspond to the 2D points. |
[in] | lambda | Regularisation parameter of the PCA shape fitting. |
[in] | num_coefficients_to_fit | How many shape-coefficients to fit (all others will stay 0). Should be bigger than zero, or boost::none to fit all coefficients. |
|
inline |
Fits the shape of a Morphable Model to given 2D landmarks (i.e. estimates the maximum likelihood solution of the shape coefficients) as proposed in [1]. It's a linear, closed-form solution fitting of the shape, with regularisation (prior towards the mean).
[1] O. Aldrian & W. Smith, Inverse Rendering of Faces with a 3D Morphable Model, PAMI 2013.
Note: Using less than the maximum number of coefficients to fit is not thoroughly tested yet and may contain an error. Note: Returns coefficients following standard normal distribution (i.e. all have similar magnitude). Why? Because we fit using the normalised basis? Note: The standard deviations given should be a vector, i.e. different for each landmark. This is not implemented yet.
[in] | morphable_model | The Morphable Model whose shape (coefficients) are estimated. |
[in] | affine_camera_matrix | A 3x4 affine camera matrix from model to screen-space (should probably be of type CV_32FC1 as all our calculations are done with float). |
[in] | landmarks | 2D landmarks from an image to fit the model to. |
[in] | vertex_ids | The vertex ids in the model that correspond to the 2D points. |
[in] | base_face | The base or reference face from where the fitting is started. Usually this would be the models mean face, which is what will be used if the parameter is not explicitly specified. |
[in] | lambda | The regularisation parameter (weight of the prior towards the mean). |
[in] | num_coefficients_to_fit | How many shape-coefficients to fit (all others will stay 0). Should be bigger than zero, or boost::none to fit all coefficients. |
[in] | detector_standard_deviation | The standard deviation of the 2D landmarks given (e.g. of the detector used), in pixels. |
[in] | model_standard_deviation | The standard deviation of the 3D vertex points in the 3D model, projected to 2D (so the value is in pixels). |
cv::Mat eos::fitting::get_3x4_affine_camera_matrix | ( | fitting::RenderingParameters | params, |
int | width, | ||
int | height | ||
) |
Creates a 3x4 affine camera matrix from given fitting parameters. The matrix transforms points directly from model-space to screen-space.
This function is mainly used since the linear shape fitting fitting::fit_shape_to_landmarks_linear expects one of these 3x4 affine camera matrices, as well as render::extract_texture.
glm::mat4x4 eos::fitting::get_4x4_modelview_matrix | ( | fitting::RenderingParameters | params | ) |
Creates a 4x4 model-view matrix from given fitting parameters.
Together with the Frustum information, this describes the full orthographic rendering parameters of the OpenGL pipeline. Example:
std::tuple<std::vector<cv::Vec2f>, std::vector<cv::Vec4f>, std::vector<int> > eos::fitting::get_contour_correspondences | ( | const eos::core::LandmarkCollection< cv::Vec2f > & | landmarks, |
const ContourLandmarks & | contour_landmarks, | ||
const ModelContour & | model_contour, | ||
float | yaw_angle, | ||
const morphablemodel::MorphableModel & | morphable_model, | ||
const glm::mat4x4 & | view_model, | ||
const glm::mat4x4 & | ortho_projection, | ||
const glm::vec4 & | viewport | ||
) |
Given a set of 2D image landmarks, finds the closest (in a L2 sense) 3D vertex from a list of vertices pre-defined in model_contour
. landmarks
can contain all landmarks, and the function will sub-select the relevant contour landmarks with the help of the given contour_landmarks
. This function choses the front-facing contour and only fits this contour to the 3D model, since these correspondences are approximately static and do not move with changing pose-angle.
It's the main contour fitting function that calls all other functions.
Note: Maybe rename to find_contour_correspondences, to highlight that there is (potentially a lot) computational cost involved? Note: Does ortho_projection have to be specifically orthographic? Otherwise, if it works with perspective too, rename to just "projection".
[in] | landmarks | All image landmarks. |
[in] | contour_landmarks | 2D image contour ids of left or right side (for example for ibug landmarks). |
[in] | model_contour | The model contour indices that should be considered to find the closest corresponding 3D vertex. |
[in] | yaw_angle | Yaw angle of the current fitting. The front-facing contour will be chosen depending on this yaw angle. |
[in] | morphable_model | A Morphable Model whose mean is used. |
[in] | view_model | Model-view matrix of the current fitting to project the 3D model vertices to 2D. |
[in] | ortho_projection | Projection matrix to project the 3D model vertices to 2D. |
[in] | viewport | Current viewport to use. |
std::tuple< std::vector< cv::Vec2f >, std::vector< cv::Vec4f >, std::vector< int > > eos::fitting::get_nearest_contour_correspondences | ( | const eos::core::LandmarkCollection< cv::Vec2f > & | landmarks, |
const std::vector< std::string > & | landmark_contour_identifiers, | ||
const std::vector< int > & | model_contour_indices, | ||
const morphablemodel::MorphableModel & | morphable_model, | ||
const glm::mat4x4 & | view_model, | ||
const glm::mat4x4 & | ortho_projection, | ||
const glm::vec4 & | viewport | ||
) |
Given a set of 2D image landmarks, finds the closest (in a L2 sense) 3D vertex from a list of vertices pre-defined in model_contour
. Assumes to be given contour correspondences of the front-facing contour.
Note: Maybe rename to find_nearest_contour_points, to highlight that there is (potentially a lot) computational cost involved? Note: Does ortho_projection have to be specifically orthographic? Otherwise, if it works with perspective too, rename to just "projection". More notes: Actually, only return the vertex id, not the point? Same with get_corresponding_pointset? Because then it's much easier to use the current shape estimate instead of the mean! But this function needs to project. So... it should take a Mesh actually? But creating a Mesh is a lot of computation? When we want to use the non-mean, then we need to use draw_sample() anyway? So overhead of Mesh is only if we use the mean? Maybe two overloads? Note: Uses the mean to calculate.
[in] | landmarks | All image landmarks. |
[in] | landmark_contour_identifiers | 2D image contour ids of left or right side (for example for ibug landmarks). |
[in] | model_contour_indices | The model contour indices that should be considered to find the closest corresponding 3D vertex. |
[in] | morphable_model | The Morphable Model whose shape (coefficients) are estimated. |
[in] | view_model | Model-view matrix of the current fitting to project the 3D model vertices to 2D. |
[in] | ortho_projection | Projection matrix to project the 3D model vertices to 2D. |
[in] | viewport | Current viewport to use. |
glm::vec4 eos::fitting::get_opencv_viewport | ( | int | width, |
int | height | ||
) |
Returns a glm/OpenGL compatible viewport vector that flips y and has the origin on the top-left, like in OpenCV.
Note: Move to detail namespace / not used at the moment.
|
inline |
Projects a point from world coordinates to screen coordinates. First, an estimated affine camera matrix is used to transform the point to clip space. Second, the point is transformed to screen coordinates using the window transform. The window transform also flips the y-axis (the image origin is top-left, while in clip space top is +1 and bottom is -1).
Note: Assumes the affine camera matrix only projects from world to clip space, because a subsequent window transform is applied. Todo: This is outdated, now that we estimate the matrix from world to screen space directly.
[in] | vertex | A vertex in 3D space. vertex[3] = 1.0f. |
[in] | affine_camera_matrix | A 3x4 affine camera matrix. |
[in] | screen_width | Width of the screen or window used for projection. |
[in] | screen_height | Height of the screen or window used for projection. |
void eos::fitting::save_rendering_parameters | ( | RenderingParameters | rendering_parameters, |
std::string | filename | ||
) |
Saves the rendering parameters for an image to a json file.
[in] | rendering_parameters | An instance of class RenderingParameters. |
[in] | filename | The file to write. |
std::runtime_error | if unable to open the given file for writing. |
std::pair< std::vector< std::string >, std::vector< int > > eos::fitting::select_contour | ( | float | yaw_angle, |
const ContourLandmarks & | contour_landmarks, | ||
const ModelContour & | model_contour | ||
) |
Takes a set of 2D and 3D contour landmarks and a yaw angle and returns two vectors with either the right or the left 2D and 3D contour indices. This function does not establish correspondence between the 2D and 3D landmarks, it just selects the front-facing contour. The two returned vectors can thus have different size. Correspondence can be established using get_nearest_contour_correspondences().
Note: Maybe rename to find_nearest_contour_points, to highlight that there is (potentially a lot) computational cost involved?
[in] | yaw_angle | yaw angle in degrees. |
[in] | contour_landmarks | 2D image contour ids of left or right side (for example for ibug landmarks). |
[in] | model_contour | The model contour indices that should be used/considered to find the closest corresponding 3D vertex. |
cv::Mat eos::fitting::to_mat | ( | const glm::mat4x4 & | glm_matrix | ) |
Converts a glm::mat4x4 to a cv::Mat.
Note: move to render namespace