Pinhole Lens
In this tutorial you will get familiar with Pinhole Lens, which is a sensor that projects 3D points on the scene
to the image plane with perspective transformation. It is the most used type of lens, as its mechanism is the same
as in the human eye or commonly used cameras.
Agenda:
- Pinhole lens model details
- PinholeLens - render and undistortion
- PinholeLens - parameters change
Scene setup
Let's use custom scene composer to set up the scene.
from skyrenderer.cases.utils import SensorSceneComposer
scene_composer = SensorSceneComposer(antialiasing_level=1)
scene_composer.setup_scene()
scene_composer.reset_camera_position([0, 0, -15])
renderer_context = scene_composer.renderer_context
2025-02-14 11:54:41,205 | skyrenderer.scene.renderer_context | INFO: Root paths:
- root path: /home/skyengine/anaconda/lib/python3.6/site-packages/skyrenderer
- assets path: /dli/mount/assets
- config path: /home/skyengine/anaconda/lib/python3.6/site-packages/skyrenderer/config
- gpu sources path: /home/skyengine/anaconda/lib/python3.6/site-packages/skyrenderer/optix_sources/sources
- cache path: /dli/mount/cache
- ptx cache path: compiled_ptx/ptx
- ocio path: ocio_configs/aces_1.2/config.ocio
Pinhole lens model details
In generalized Lens projection model (INTRO_Lens), following re-projection steps are done:
- (u,v) are reprojected onto projection plane (z=1).
- Points are being undistorted, to match original ray directions (using distortion back mapping and
precalculated distortion map). - Based on re-projection radial distance and projection equation ray angle (theta) is calculated.
- Projection sphere coordinates are reconstructed (unit vector with angle theta between incident plane pi and
sensor optical axis). - Ray made from O and P' is generated.
Pinhole camera distortion (2nd step)
In real cameras, lenses introduce distortion, especially radial and tangential distortion, which can affect the
image. These distortions are most apparent at the edges of the image and arise due to the physical properties of
the lens used in the camera.
Distorted coordinates of a point are transformed with radial based and tangential distortion as follows:
\begin{equation}
\begin{bmatrix}
u \
v
\end{bmatrix} = \begin{bmatrix}
f_x x' + c_x \
f_y y' + c_y
\end{bmatrix}
\end{equation}
where
\begin{align}
\begin{bmatrix}
x' \
y'
\end{bmatrix} &= \begin{bmatrix}
x \cdot r_f + 2p_1 \cdot x \cdot y + p_2 \cdot (r^2+2x^2) + s_1 \cdot r^2 + s_2 \cdot r^4 \
y \cdot r_f + p_1 \cdot (r^2+2y^2) + 2p_2 \cdot x \cdot y + s_3 \cdot r^2 + s_4 \cdot r^4 \
\end{bmatrix} \
r_f &= \dfrac{1+k_1\cdot r^2+k_2\cdot r^4+k_3\cdot r^6}{1+k_4\cdot r^2+k_5\cdot r^4+k_6\cdot r^6} \
r &= x^2+y^2 \
\begin{bmatrix}
x\
y
\end{bmatrix} &= \begin{bmatrix}
X_c/Z_c \
Y_c/Z_c
\end{bmatrix} \
\end{align}
Pinhole re-projection (3rd step)
In pinhole model incident angle and sensor matrix radius are connected with each other according to equation:
\begin{align}
r &= f \cdot \tan{\theta} \to r = \tan \theta, & \text{when focal length is normalized} \
\end{align}
Camera extrinsic parameters
Extrinsic parameters describe the camera's position and orientation in the 3D world. They consist of:
- Rotation ($R$): The orientation of the camera relative to the world.
- Translation ($T$): The position of the camera center relative to the world origin.
The extrinsic parameters form the 3x4 camera extrinsic matrix $[R|T]$, which transforms world coordinates into
camera coordinates. They are stored in SceneLayoutNode/SceneNode transformations.
Camera intrinsic parameters
Intrinsic parameters stores the properties of the camera itself that affect how 3D points are mapped to
2D coordinates. They include:
- Focal length ($f$) - the distance from the pinhole to the image plane.
- Principal point ($c_x, c_y$) - the point where the principal axis intersects the image plane (often the image
center).
Intrinsic parameters are passed in the lens parameters. More about the influence of intrinsic parameters on ray
generation can be found in INTRO_Lens.py tutorial.
PinholeLens - render and undistortion
In our implementation of the PinholeLens, we can adjust parameters presented in the section above.
In this example, we will adjust camera focal length and principal point, as well as all distortions parameters.
PinholeLens setup and visualization
# Image resolution
width = 1000
height = 1000
# Camera intrinsic parameters
camera_fx = 750
camera_fy = 750
camera_cx = width / 2
camera_cy = height / 2
# Camera distortion parameters
dist_k1 = -0.2
dist_k2 = 0.1660696363841201
dist_k3 = -0.1046838971475154
dist_k4 = -0.05589875277472606
dist_k5 = 0
dist_k6 = 0
dist_p1 = -0.00104636649317819
dist_p2 = -0.00168027284332261
dist_s1 = 0.01253
dist_s2 = 0.05434
dist_s3 = 0.08452
dist_s4 = 0.01137
dist_sampling_factor = 4
dist_range_extension = 0
We can pass the above parameters to the PinholeLens through parameter provider, and pass parametrized lens to
VisibleLightRenderStep as in SENSOR_PinholeLens tutorial.
from skyrenderer.render_chain import RenderChain, VisibleLightRenderStep, Denoiser
from skyrenderer.render_chain.camera_steps.Lens.pinhole_lens import PinholeLens
pinhole_params = PinholeLens.create_parameter_provider(
renderer_context,
fx=camera_fx,
fy=camera_fy,
cx_relative=0.5,
cy_relative=0.5,
dist_k1=dist_k1,
dist_k2=dist_k2,
dist_k3=dist_k3,
dist_k4=dist_k4,
dist_k5=dist_k5,
dist_k6=dist_k6,
dist_p1=dist_p1,
dist_p2=dist_p2,
dist_s1=dist_s1,
dist_s2=dist_s2,
dist_s3=dist_s3,
dist_s4=dist_s4,
dist_sampling_factor=dist_sampling_factor,
dist_range_extension=dist_range_extension,
)
lens = PinholeLens(renderer_context, parameter_provider=pinhole_params)
rs = VisibleLightRenderStep(
renderer_context,
lens=lens,
origin_name="camera_CAM_NUL",
target_name="top_node",
)
renderer_context.define_render_chain(
RenderChain(render_steps=[rs, Denoiser(renderer_context)], width=width, height=height)
)
distorted_image = scene_composer.get_render()
scene_composer.visualize(distorted_image)
2025-02-14 11:54:44,228 | skyrenderer.utils.time_measurement | INFO: Setup time: 2.97 seconds
2025-02-14 11:54:44,755 | skyrenderer.utils.time_measurement | INFO: Context update time: 526 ms
2025-02-14 11:54:45,472 | skyrenderer.utils.time_measurement | INFO: Key points calculation time: 0 ms
2025-02-14 11:54:45,472 | skyrenderer.utils.time_measurement | INFO: Render time: 716 ms
Image undistortion
Our Pinhole model implementation is compatible with the state-of-the-art solutions (e.g. OpenCV), therefore we can
perform undistortion on the rendered image with 3rd party library:
import cv2 as cv
import numpy as np
camera_matrix = np.array([[camera_fx, 0, camera_cx], [0, camera_fy, camera_cy], [0, 0, 1]], np.float)
dist_coeffs = np.array(
[dist_k1, dist_k2, dist_p1, dist_p2, dist_k3, dist_k4, dist_k5, dist_k6, dist_s1, dist_s2, dist_s3, dist_s4],
np.float,
)
img_undistorted = cv.undistort(distorted_image, camera_matrix, dist_coeffs)
scene_composer.visualize(img_undistorted)
PinholeLens - parameters change
fx, fy - focal length
width = 450
height = 450
renders = {}
scene_composer.setup_pinhole_lens(width, height, fx=200)
renders["fx=200"] = scene_composer.get_render()
scene_composer.setup_pinhole_lens(width, height, fx=915)
renders["fx=915(default)"] = scene_composer.get_render()
scene_composer.setup_pinhole_lens(width, height, fx=1500)
renders["fx=1500"] = scene_composer.get_render()
scene_composer.setup_pinhole_lens(width, height, fx=450, fy=200)
renders["fy=200"] = scene_composer.get_render()
scene_composer.setup_pinhole_lens(width, height, fy=915)
renders["fy=915(default)"] = scene_composer.get_render()
scene_composer.setup_pinhole_lens(width, height, fy=1500)
renders["fy=1500"] = scene_composer.get_render()
scene_composer.setup_pinhole_lens(width, height, fx=200, fy=200)
renders["fx=fy=200"] = scene_composer.get_render()
scene_composer.setup_pinhole_lens(width, height, fx=915, fy=915)
renders["fx=fy=915(default)"] = scene_composer.get_render()
scene_composer.setup_pinhole_lens(width, height, fx=1500, fy=1500)
renders["fx=fy=1500"] = scene_composer.get_render()
scene_composer.visualize_grid_desc(renders, shape=(3 height, 3 width), n_cols=3, font_scale=2)
2025-02-14 11:54:46,301 | skyrenderer.utils.time_measurement | INFO: Setup time: 180 ms
2025-02-14 11:54:46,785 | skyrenderer.utils.time_measurement | INFO: Context update time: 484 ms
2025-02-14 11:54:46,957 | skyrenderer.utils.time_measurement | INFO: Key points calculation time: 0 ms
2025-02-14 11:54:46,958 | skyrenderer.utils.time_measurement | INFO: Render time: 172 ms
2025-02-14 11:54:47,004 | skyrenderer.utils.time_measurement | INFO: Setup time: 43 ms
2025-02-14 11:54:47,483 | skyrenderer.utils.time_measurement | INFO: Context update time: 478 ms
2025-02-14 11:54:48,363 | skyrenderer.utils.time_measurement | INFO: Key points calculation time: 0 ms
2025-02-14 11:54:48,363 | skyrenderer.utils.time_measurement | INFO: Render time: 880 ms
2025-02-14 11:54:48,410 | skyrenderer.utils.time_measurement | INFO: Setup time: 42 ms
2025-02-14 11:54:48,877 | skyrenderer.utils.time_measurement | INFO: Context update time: 466 ms
2025-02-14 11:54:49,102 | skyrenderer.utils.time_measurement | INFO: Key points calculation time: 0 ms
2025-02-14 11:54:49,102 | skyrenderer.utils.time_measurement | INFO: Render time: 225 ms
2025-02-14 11:54:50,152 | skyrenderer.utils.time_measurement | INFO: Setup time: 1.05 seconds
2025-02-14 11:54:50,655 | skyrenderer.utils.time_measurement | INFO: Context update time: 502 ms
2025-02-14 11:54:51,519 | skyrenderer.utils.time_measurement | INFO: Key points calculation time: 0 ms
2025-02-14 11:54:51,520 | skyrenderer.utils.time_measurement | INFO: Render time: 865 ms
2025-02-14 11:54:51,577 | skyrenderer.utils.time_measurement | INFO: Setup time: 52 ms
2025-02-14 11:54:52,088 | skyrenderer.utils.time_measurement | INFO: Context update time: 510 ms
2025-02-14 11:54:52,264 | skyrenderer.utils.time_measurement | INFO: Key points calculation time: 0 ms
2025-02-14 11:54:52,264 | skyrenderer.utils.time_measurement | INFO: Render time: 176 ms
2025-02-14 11:54:52,312 | skyrenderer.utils.time_measurement | INFO: Setup time: 43 ms
2025-02-14 11:54:52,823 | skyrenderer.utils.time_measurement | INFO: Context update time: 511 ms
2025-02-14 11:54:54,005 | skyrenderer.utils.time_measurement | INFO: Key points calculation time: 0 ms
2025-02-14 11:54:54,005 | skyrenderer.utils.time_measurement | INFO: Render time: 1.18 seconds
2025-02-14 11:54:54,055 | skyrenderer.utils.time_measurement | INFO: Setup time: 45 ms
2025-02-14 11:54:54,523 | skyrenderer.utils.time_measurement | INFO: Context update time: 467 ms
2025-02-14 11:54:54,688 | skyrenderer.utils.time_measurement | INFO: Key points calculation time: 0 ms
2025-02-14 11:54:54,689 | skyrenderer.utils.time_measurement | INFO: Render time: 165 ms
2025-02-14 11:54:54,736 | skyrenderer.utils.time_measurement | INFO: Setup time: 43 ms
2025-02-14 11:54:55,239 | skyrenderer.utils.time_measurement | INFO: Context update time: 503 ms
2025-02-14 11:54:55,416 | skyrenderer.utils.time_measurement | INFO: Key points calculation time: 0 ms
2025-02-14 11:54:55,417 | skyrenderer.utils.time_measurement | INFO: Render time: 177 ms
2025-02-14 11:54:55,464 | skyrenderer.utils.time_measurement | INFO: Setup time: 43 ms
2025-02-14 11:54:55,951 | skyrenderer.utils.time_measurement | INFO: Context update time: 486 ms
2025-02-14 11:54:56,125 | skyrenderer.utils.time_measurement | INFO: Key points calculation time: 0 ms
2025-02-14 11:54:56,126 | skyrenderer.utils.time_measurement | INFO: Render time: 175 ms
cx_relative, cy_relative - center of projection
renders = {}
scene_composer.setup_pinhole_lens(width, height, cx_relative=0.2)
renders["cx=0.2"] = scene_composer.get_render()
scene_composer.setup_pinhole_lens(width, height, cx_relative=0.5)
renders["cx=0.5(default)"] = scene_composer.get_render()
scene_composer.setup_pinhole_lens(width, height, cx_relative=0.8)
renders["cx=0.8"] = scene_composer.get_render()
scene_composer.setup_pinhole_lens(width, height, cx_relative=0.5, cy_relative=0.2)
renders["cy=0.2"] = scene_composer.get_render()
scene_composer.setup_pinhole_lens(width, height, cy_relative=0.5)
renders["cy=0.5(default)"] = scene_composer.get_render()
scene_composer.setup_pinhole_lens(width, height, cy_relative=0.8)
renders["cy=0.8"] = scene_composer.get_render()
scene_composer.setup_pinhole_lens(width, height, cx_relative=0.2, cy_relative=0.2)
renders["cx=cy=0.2"] = scene_composer.get_render()
scene_composer.setup_pinhole_lens(width, height, cx_relative=0.5, cy_relative=0.5)
renders["cx=cy=0.5(default)"] = scene_composer.get_render()
scene_composer.setup_pinhole_lens(width, height, cx_relative=0.8, cy_relative=0.8)
renders["cx=cy=0.8"] = scene_composer.get_render()
scene_composer.visualize_grid_desc(renders, shape=(3 height, 3 width), n_cols=3, font_scale=2)
2025-02-14 11:54:56,763 | skyrenderer.utils.time_measurement | INFO: Setup time: 44 ms
2025-02-14 11:54:57,261 | skyrenderer.utils.time_measurement | INFO: Context update time: 497 ms
2025-02-14 11:54:58,202 | skyrenderer.utils.time_measurement | INFO: Key points calculation time: 0 ms
2025-02-14 11:54:58,203 | skyrenderer.utils.time_measurement | INFO: Render time: 941 ms
2025-02-14 11:54:58,251 | skyrenderer.utils.time_measurement | INFO: Setup time: 44 ms
2025-02-14 11:54:58,754 | skyrenderer.utils.time_measurement | INFO: Context update time: 503 ms
2025-02-14 11:54:59,782 | skyrenderer.utils.time_measurement | INFO: Key points calculation time: 0 ms
2025-02-14 11:54:59,783 | skyrenderer.utils.time_measurement | INFO: Render time: 1.03 seconds
2025-02-14 11:54:59,829 | skyrenderer.utils.time_measurement | INFO: Setup time: 43 ms
2025-02-14 11:55:00,387 | skyrenderer.utils.time_measurement | INFO: Context update time: 556 ms
2025-02-14 11:55:01,388 | skyrenderer.utils.time_measurement | INFO: Key points calculation time: 0 ms
2025-02-14 11:55:01,388 | skyrenderer.utils.time_measurement | INFO: Render time: 1.00 seconds
2025-02-14 11:55:01,436 | skyrenderer.utils.time_measurement | INFO: Setup time: 43 ms
2025-02-14 11:55:01,970 | skyrenderer.utils.time_measurement | INFO: Context update time: 533 ms
2025-02-14 11:55:02,992 | skyrenderer.utils.time_measurement | INFO: Key points calculation time: 0 ms
2025-02-14 11:55:02,992 | skyrenderer.utils.time_measurement | INFO: Render time: 1.02 seconds
2025-02-14 11:55:03,040 | skyrenderer.utils.time_measurement | INFO: Setup time: 43 ms
2025-02-14 11:55:03,559 | skyrenderer.utils.time_measurement | INFO: Context update time: 519 ms
2025-02-14 11:55:04,542 | skyrenderer.utils.time_measurement | INFO: Key points calculation time: 0 ms
2025-02-14 11:55:04,543 | skyrenderer.utils.time_measurement | INFO: Render time: 983 ms
2025-02-14 11:55:04,592 | skyrenderer.utils.time_measurement | INFO: Setup time: 45 ms
2025-02-14 11:55:05,087 | skyrenderer.utils.time_measurement | INFO: Context update time: 494 ms
2025-02-14 11:55:06,090 | skyrenderer.utils.time_measurement | INFO: Key points calculation time: 0 ms
2025-02-14 11:55:06,091 | skyrenderer.utils.time_measurement | INFO: Render time: 1.00 seconds
2025-02-14 11:55:06,138 | skyrenderer.utils.time_measurement | INFO: Setup time: 43 ms
2025-02-14 11:55:06,634 | skyrenderer.utils.time_measurement | INFO: Context update time: 496 ms
2025-02-14 11:55:07,603 | skyrenderer.utils.time_measurement | INFO: Key points calculation time: 0 ms
2025-02-14 11:55:07,604 | skyrenderer.utils.time_measurement | INFO: Render time: 969 ms
2025-02-14 11:55:07,651 | skyrenderer.utils.time_measurement | INFO: Setup time: 43 ms
2025-02-14 11:55:08,155 | skyrenderer.utils.time_measurement | INFO: Context update time: 503 ms
2025-02-14 11:55:09,200 | skyrenderer.utils.time_measurement | INFO: Key points calculation time: 0 ms
2025-02-14 11:55:09,201 | skyrenderer.utils.time_measurement | INFO: Render time: 1.05 seconds
2025-02-14 11:55:09,248 | skyrenderer.utils.time_measurement | INFO: Setup time: 43 ms
2025-02-14 11:55:09,737 | skyrenderer.utils.time_measurement | INFO: Context update time: 488 ms
2025-02-14 11:55:09,917 | skyrenderer.utils.time_measurement | INFO: Key points calculation time: 0 ms
2025-02-14 11:55:09,918 | skyrenderer.utils.time_measurement | INFO: Render time: 181 ms
aperture_jitter and focus_distance
apperture_jitter is the parameter responsible for aperture related to focal length. The higher value, the smaller
DOF of a camera. If 0, infinite DOF is present. On the other hand, focus_distance is parameter holding a value of
plane distance in meters. It is used, when aperture jitter is nonzero.
renders = {}
scene_composer.setup_pinhole_lens(width, height, cx_relative=0.5, cy_relative=0.5)
renders["aperture=0(default)"] = scene_composer.get_render()
scene_composer.setup_pinhole_lens(width, height, aperture_jitter=1, focus_distance=5)
renders["aperture=1,focus_dist=5"] = scene_composer.get_render()
scene_composer.setup_pinhole_lens(width, height, aperture_jitter=1, focus_distance=15)
renders["aperture=1,focus_dist=15"] = scene_composer.get_render()
scene_composer.setup_pinhole_lens(width, height, aperture_jitter=1, focus_distance=25)
renders["aperture=1,focus_dist=25"] = scene_composer.get_render()
scene_composer.visualize_grid_desc(
renders, shape=(height, 4 * width), n_cols=4, font_scale=1, text_position=(20, 20)
)
2025-02-14 11:55:10,666 | skyrenderer.utils.time_measurement | INFO: Setup time: 43 ms
2025-02-14 11:55:11,150 | skyrenderer.utils.time_measurement | INFO: Context update time: 483 ms
2025-02-14 11:55:11,327 | skyrenderer.utils.time_measurement | INFO: Key points calculation time: 0 ms
2025-02-14 11:55:11,328 | skyrenderer.utils.time_measurement | INFO: Render time: 177 ms
2025-02-14 11:55:11,376 | skyrenderer.utils.time_measurement | INFO: Setup time: 43 ms
2025-02-14 11:55:11,879 | skyrenderer.utils.time_measurement | INFO: Context update time: 502 ms
2025-02-14 11:55:12,894 | skyrenderer.utils.time_measurement | INFO: Key points calculation time: 0 ms
2025-02-14 11:55:12,895 | skyrenderer.utils.time_measurement | INFO: Render time: 1.02 seconds
2025-02-14 11:55:12,942 | skyrenderer.utils.time_measurement | INFO: Setup time: 42 ms
2025-02-14 11:55:13,453 | skyrenderer.utils.time_measurement | INFO: Context update time: 510 ms
2025-02-14 11:55:14,461 | skyrenderer.utils.time_measurement | INFO: Key points calculation time: 0 ms
2025-02-14 11:55:14,461 | skyrenderer.utils.time_measurement | INFO: Render time: 1.01 seconds
2025-02-14 11:55:14,508 | skyrenderer.utils.time_measurement | INFO: Setup time: 43 ms
2025-02-14 11:55:15,006 | skyrenderer.utils.time_measurement | INFO: Context update time: 497 ms
2025-02-14 11:55:15,182 | skyrenderer.utils.time_measurement | INFO: Key points calculation time: 0 ms
2025-02-14 11:55:15,183 | skyrenderer.utils.time_measurement | INFO: Render time: 176 ms
Summary
In this section you have learnt:
- All parameters from pinhole lens mathematical theory can be set in PinholeLens's
create_parameter_provider
method. - Pinhole lens model follows generic projection model described in INTRO_Lens.
- Model-based parameters of pinhole lens are compatible with popular CV libraries like OpenCV, and renders can be
undistorted using 3rd party library.