eos  0.7.1
Classes | Enumerations | Functions
eos::render Namespace Reference

3D mesh and texture extraction functionality. More...

Classes

struct  Mesh
 This class represents a 3D mesh consisting of vertices, vertex colour information and texture coordinates. More...
 
class  Texture
 Represents a texture for rendering. More...
 

Enumerations

enum  TextureInterpolation { NearestNeighbour, Bilinear, Area }
 

Functions

void write_obj (Mesh mesh, std::string filename)
 Writes the given Mesh to an obj file that for example can be read by MeshLab. More...
 
void write_textured_obj (Mesh mesh, std::string filename)
 Writes an obj file of the given Mesh, including texture coordinates, and an mtl file containing a reference to the isomap. More...
 
std::pair< cv::Mat, cv::Mat > render (Mesh mesh, cv::Mat model_view_matrix, cv::Mat projection_matrix, int viewport_width, int viewport_height, const boost::optional< Texture > &texture=boost::none, bool enable_backface_culling=false, bool enable_near_clipping=true, bool enable_far_clipping=true)
 
std::pair< cv::Mat, cv::Mat > render_affine (Mesh mesh, cv::Mat affine_camera_matrix, int viewport_width, int viewport_height, bool do_backface_culling=true)
 
cv::Mat extract_texture (Mesh mesh, cv::Mat affine_camera_matrix, cv::Mat image, cv::Mat depthbuffer, bool compute_view_angle, TextureInterpolation mapping_type, int isomap_resolution)
 
cv::Mat extract_texture (Mesh mesh, cv::Mat affine_camera_matrix, cv::Mat image, bool compute_view_angle=false, TextureInterpolation mapping_type=TextureInterpolation::NearestNeighbour, int isomap_resolution=512)
 
cv::Vec2f clip_to_screen_space (const cv::Vec2f &clip_coordinates, int screen_width, int screen_height)
 
cv::Vec2f screen_to_clip_space (const cv::Vec2f &screen_coordinates, int screen_width, int screen_height)
 
cv::Vec3f calculate_face_normal (const cv::Vec3f &v0, const cv::Vec3f &v1, const cv::Vec3f &v2)
 
cv::Mat draw_texcoords (Mesh mesh, cv::Mat image=cv::Mat())
 
unsigned int get_max_possible_mipmaps_num (unsigned int width, unsigned int height)
 
bool is_power_of_two (int x)
 
Texture create_mipmapped_texture (cv::Mat image, unsigned int mipmapsNum=0)
 

Detailed Description

3D mesh and texture extraction functionality.

Enumeration Type Documentation

The interpolation types that can be used to map the texture from the original image to the isomap.

Function Documentation

cv::Vec3f eos::render::calculate_face_normal ( const cv::Vec3f &  v0,
const cv::Vec3f &  v1,
const cv::Vec3f &  v2 
)

Calculates the normal of a face (or triangle), i.e. the per-face normal. Return normal will be normalised. Assumes the triangle is given in CCW order, i.e. vertices in counterclockwise order on the screen are front-facing.

Parameters
[in]v0First vertex.
[in]v1Second vertex.
[in]v2Third vertex.
Returns
The unit-length normal of the given triangle.
cv::Vec2f eos::render::clip_to_screen_space ( const cv::Vec2f &  clip_coordinates,
int  screen_width,
int  screen_height 
)
inline

Transforms a point from clip space ([-1, 1] x [-1, 1]) to image (screen) coordinates, i.e. the window transform. Note that the y-coordinate is flipped because the image origin is top-left while in clip space top is +1 and bottom is -1. No z-division is performed. Note: It should rather be called from NDC to screen space?

Exactly conforming to the OpenGL viewport transform, except that we flip y at the end. Qt: Origin top-left. OpenGL: bottom-left. OCV: top-left.

Parameters
[in]clip_coordinatesA point in clip coordinates.
[in]screen_widthWidth of the screen or window.
[in]screen_heightHeight of the screen or window.
Returns
A vector with x and y coordinates transformed to screen space.
cv::Mat eos::render::draw_texcoords ( Mesh  mesh,
cv::Mat  image = cv::Mat() 
)

Draws the texture coordinates (uv-coords) of the given mesh into an image by looping over the triangles and drawing each triangle's texcoords.

Parameters
[in]meshA mesh with texture coordinates.
[in]imageAn optional image to draw onto.
Returns
An image with the texture coordinate triangles drawn in it, 512x512 if no image is given.
cv::Mat eos::render::extract_texture ( Mesh  mesh,
cv::Mat  affine_camera_matrix,
cv::Mat  image,
cv::Mat  depthbuffer,
bool  compute_view_angle = false,
TextureInterpolation  mapping_type = TextureInterpolation::NearestNeighbour,
int  isomap_resolution = 512 
)
inline

Extracts the texture of the face from the given image and stores it as isomap (a rectangular texture map). This function can be used if a depth buffer has already been computed. To just run the texture extraction, see the overload extract_texture(Mesh, cv::Mat, cv::Mat, TextureInterpolation, int).

It might be wise to remove this overload as it can get quite confusing with the zbuffer. Obviously the depthbuffer given should have been created with the same (affine or ortho) projection matrix than the texture extraction is called with.

Parameters
[in]meshA mesh with texture coordinates.
[in]affine_camera_matrixAn estimated 3x4 affine camera matrix.
[in]imageThe image to extract the texture from.
[in]depthbufferA pre-calculated depthbuffer image.
[in]compute_view_angleA flag whether the view angle of each vertex should be computed and returned. If set to true, the angle will be encoded into the alpha channel (0 meaning occluded or facing away 90°, 127 meaning facing a 45° angle and 255 meaning front-facing, and all values in between). If set to false, the alpha channel will only contain 0 for occluded vertices and 255 for visible vertices.
[in]mapping_typeThe interpolation type to be used for the extraction.
[in]isomap_resolutionThe resolution of the generated isomap. Defaults to 512x512.
Returns
The extracted texture as isomap (texture map).
cv::Mat eos::render::extract_texture ( Mesh  mesh,
cv::Mat  affine_camera_matrix,
cv::Mat  image,
bool  compute_view_angle = false,
TextureInterpolation  mapping_type = TextureInterpolation::NearestNeighbour,
int  isomap_resolution = 512 
)
inline

Extracts the texture of the face from the given image and stores it as isomap (a rectangular texture map).

Note/Todo: Only use TextureInterpolation::NearestNeighbour for the moment, the other methods don't have correct handling of the alpha channel (and will most likely throw an exception).

Todo: These should be renamed to extract_texture_affine? Can we combine both cases somehow? Or an overload with RenderingParameters?

For TextureInterpolation::NearestNeighbour, returns a 4-channel isomap with the visibility in the 4th channel (0=invis, 255=visible).

Parameters
[in]meshA mesh with texture coordinates.
[in]affine_camera_matrixAn estimated 3x4 affine camera matrix.
[in]imageThe image to extract the texture from. Should be 8UC3, other types not supported yet.
[in]compute_view_angleA flag whether the view angle of each vertex should be computed and returned. If set to true, the angle will be encoded into the alpha channel (0 meaning occluded or facing away 90°, 127 meaning facing a 45° angle and 255 meaning front-facing, and all values in between). If set to false, the alpha channel will only contain 0 for occluded vertices and 255 for visible vertices.
[in]mapping_typeThe interpolation type to be used for the extraction.
[in]isomap_resolutionThe resolution of the generated isomap. Defaults to 512x512.
Returns
The extracted texture as isomap (texture map).
std::pair<cv::Mat, cv::Mat> eos::render::render ( Mesh  mesh,
cv::Mat  model_view_matrix,
cv::Mat  projection_matrix,
int  viewport_width,
int  viewport_height,
const boost::optional< Texture > &  texture = boost::none,
bool  enable_backface_culling = false,
bool  enable_near_clipping = true,
bool  enable_far_clipping = true 
)

This file implements a software renderer conforming to OpenGL conventions. The following are implementation notes, mostly for reference, or as a reminder of what exactly is going on. Don't try to understand them :-)

The renderer was initially based on code by Wojciech Sterna (http://maxest.gct-game.net/content/vainmoinen/index.html), however, it has since then been completely rewritten. Still I'd like to thank him for making his code available and bravely answering my questions via email.

Coordinate systems: When specifying the vertices: +x = right, +y = up, we look into -z. So z = 0.5 is in front of 0.0. Z-Buffer:

Shirley: Specify n and f with negative values. which makes sense b/c the points are along the -z axis. Consequences: notably: orthogonal(2, 3): Shirley has denominator (n-f). In what space are the points in Shirley after this? OGL: We're in the orthographic viewing volume looking down -z. However, n and f are specified positive.

B/c the 3D points in front of the cam obviously still have negative z values, the z-value is negated. So: n = 0.1, f = 100; With the given OpenGL ortho matrix, it means a point on the near-plane which will have z = -0.1 will land up on z_clip (which equals z_ndc with ortho because w=1) = -1, and a point on the far plane z = -100 will have z_ndc = +1.

That's also why in the perspective case, w_clip is set to -z_eye because to project a point the formula is $x_p = (-n * x_e)/z_e$ (because our near is specified with positive values, but the near-plane is really at -n); but now we just move the minus-sign to the denominator, $x_p = (n * x_e)/-z_e$, so in the projection matrix we can use the (positive) n and f values and afterwards we divide by w = -z_e.

http://www.songho.ca/opengl/gl_projectionmatrix.html

Random notes: clip-space: after applying the projection matrix. ndc: after division by w NDC cube: the range of x-coordinate from [l, r] to [-1, 1], the y-coordinate from [b, t] to [-1, 1] and the z-coordinate from [n, f] to [-1, 1].

Note/Todo: I read that in screen space, OpenGL transform the z-values again to be between 0 and 1?

In contrast to OGL, this renderer doesn't have state, it's just a function that gets called with all necessary parameters. It's easiest for our purposes.

Here's the whole rendering pipeline: Model space -> model transforms World space -> camera (view/eye) transform View / eye / camera space ("truncated pyramid frustum". In case of ortho, it's already rectangular.) -> perspective/ortho projection Clip coords (x_c, y_c, z_c, w_c); the z-axis is flipped now. z [z=-n, z=-f] is mapped to [-1, +1] in case of ortho, but not yet in case of persp (it's also flipped though), but the not-[-1,1]-range is fine as we test against w_c. I.e. the larger the z-value, the further back we are. Do frustum culling (clipping) here. Test the clip-coords with w_c, and discard if a tri is completely outside. Of the partially visible tris, clip them against the near-plane and construct the visible part of the triangle. We only do this for the near-plane here. Clipping to the near plane must be done here because after w-division triangles crossing it would get distorted. "Then, OpenGL will reconstruct the edges of the polygon where clipping occurs." -> Then divide by the w component of the clip coordinates NDC. (now only 3D vectors: [x_ndc, y_ndc, z_ndc]). nearest points have z=-1, points on far plane have z=+1. -> window transform. (also, OGL does some more to the z-buffer?) Screen / window space Directly after window-transform (still processing triangles), do backface culling with areVerticesCCWInScreenSpace() Directly afterwards we calculate the triangle's bounding box and clip x/y (screen) against 0 and the viewport width/height. Rasterising: Clipping against the far plane here by only drawing those pixels with a z-value of <= 1.0f.

OGL: "both clipping (frustum culling) and NDC transformations are integrated into GL_PROJECTION matrix"

Note: In both the ortho and persp case, points at z=-n end up at -1, z=-f at +1. In case of persp proj., this happens only after the divide by w. Renders the given mesh onto a 2D image using 4x4 model-view and projection matrices. Conforms to OpenGL conventions.

Parameters
[in]meshA 3D mesh.
[in]model_view_matrixA 4x4 OpenGL model-view matrix.
[in]projection_matrixA 4x4 orthographic or perspective OpenGL projection matrix.
[in]viewport_widthScreen width.
[in]viewport_heightScreen height.
[in]textureAn optional texture map. If not given, vertex-colouring is used.
[in]enable_backface_cullingWhether the renderer should perform backface culling. If true, only draw triangles with vertices ordered CCW in screen-space.
[in]enable_near_clippingWhether vertices should be clipped against the near plane.
[in]enable_far_clippingWhether vertices should be clipped against the far plane.
Returns
A pair with the colourbuffer as its first element and the depthbuffer as the second element.
std::pair<cv::Mat, cv::Mat> eos::render::render_affine ( Mesh  mesh,
cv::Mat  affine_camera_matrix,
int  viewport_width,
int  viewport_height,
bool  do_backface_culling = true 
)

Renders the mesh using the given affine camera matrix and returns the colour and depth buffer images. The camera matrix should be one estimated with fitting::estimate_affine_camera (Hartley & Zisserman algorithm).

If the given mesh is a shape-only mesh without vertex-colour information, the vertices will be rendered in grey.

#Todo: May consider an overload where we pass in an image, use that as colourbuffer and draw over it. #Todo: Add texture rendering to this. Then, create an additional function in extract_texure that is fully optimised for only the extraction.

Parameters
[in]meshA 3D mesh.
[in]affine_camera_matrix3x4 affine camera matrix.
[in]viewport_widthScreen width.
[in]viewport_heightScreen height.
[in]do_backface_cullingWhether the renderer should perform backface culling.
Returns
A pair with the colourbuffer as its first element and the depthbuffer as the second element.
cv::Vec2f eos::render::screen_to_clip_space ( const cv::Vec2f &  screen_coordinates,
int  screen_width,
int  screen_height 
)
inline

Transforms a point from image (screen) coordinates to clip space ([-1, 1] x [-1, 1]). Note that the y-coordinate is flipped because the image origin is top-left while in clip space top is +1 and bottom is -1.

Parameters
[in]screen_coordinatesA point in screen coordinates.
[in]screen_widthWidth of the screen or window.
[in]screen_heightHeight of the screen or window.
Returns
A vector with x and y coordinates transformed to clip space.
void eos::render::write_obj ( Mesh  mesh,
std::string  filename 
)
inline

Writes the given Mesh to an obj file that for example can be read by MeshLab.

If the mesh contains vertex colour information, it will be written to the obj as well.

Parameters
[in]meshThe mesh to save as obj.
[in]filenameOutput filename (including ".obj").
void eos::render::write_textured_obj ( Mesh  mesh,
std::string  filename 
)
inline

Writes an obj file of the given Mesh, including texture coordinates, and an mtl file containing a reference to the isomap.

The obj will contain texture coordinates for the mesh, and the mtl file will link to a file named <filename>.isomap.png. Note that the texture (isomap) has to be saved separately.

Parameters
[in]meshThe mesh to save as obj.
[in]filenameOutput filename.