• lens

Pinhole lens parameters

By: SKY ENGINE AI
scroll down ↓to find out morepinhole-lens-parameters_1_resourcesTutorial

Pinhole Lens

In this tutorial you will get familiar with Pinhole Lens, which is a sensor that projects 3D points on the scene
to the image plane with perspective transformation. It is the most used type of lens, as its mechanism is the same
as in the human eye or commonly used cameras.

Agenda:

  • Pinhole lens model details
  • PinholeLens - render and undistortion
  • PinholeLens - parameters change

Scene setup

Let's use custom scene composer to set up the scene.

from skyrenderer.cases.utils import SensorSceneComposer scene_composer = SensorSceneComposer(antialiasing_level=1) scene_composer.setup_scene() scene_composer.reset_camera_position([0, 0, -15]) renderer_context = scene_composer.renderer_context
2025-02-14 11:54:41,205 | skyrenderer.scene.renderer_context | INFO: Root paths: - root path: /home/skyengine/anaconda/lib/python3.6/site-packages/skyrenderer - assets path: /dli/mount/assets - config path: /home/skyengine/anaconda/lib/python3.6/site-packages/skyrenderer/config - gpu sources path: /home/skyengine/anaconda/lib/python3.6/site-packages/skyrenderer/optix_sources/sources - cache path: /dli/mount/cache - ptx cache path: compiled_ptx/ptx - ocio path: ocio_configs/aces_1.2/config.ocio

Pinhole lens model details

In generalized Lens projection model (INTRO_Lens), following re-projection steps are done:

  1. (u,v) are reprojected onto projection plane (z=1).
  2. Points are being undistorted, to match original ray directions (using distortion back mapping and
    precalculated distortion map).
  3. Based on re-projection radial distance and projection equation ray angle (theta) is calculated.
  4. Projection sphere coordinates are reconstructed (unit vector with angle theta between incident plane pi and
    sensor optical axis).
  5. Ray made from O and P' is generated.

Pinhole camera distortion (2nd step)

In real cameras, lenses introduce distortion, especially radial and tangential distortion, which can affect the
image. These distortions are most apparent at the edges of the image and arise due to the physical properties of
the lens used in the camera.
Distorted coordinates of a point are transformed with radial based and tangential distortion as follows:
\begin{equation}
\begin{bmatrix}
u \
v
\end{bmatrix} = \begin{bmatrix}
f_x x' + c_x \
f_y y' + c_y
\end{bmatrix}
\end{equation}

where

\begin{align}
\begin{bmatrix}
x' \
y'
\end{bmatrix} &= \begin{bmatrix}
x \cdot r_f + 2p_1 \cdot x \cdot y + p_2 \cdot (r^2+2x^2) + s_1 \cdot r^2 + s_2 \cdot r^4 \
y \cdot r_f + p_1 \cdot (r^2+2y^2) + 2p_2 \cdot x \cdot y + s_3 \cdot r^2 + s_4 \cdot r^4 \
\end{bmatrix} \
r_f &= \dfrac{1+k_1\cdot r^2+k_2\cdot r^4+k_3\cdot r^6}{1+k_4\cdot r^2+k_5\cdot r^4+k_6\cdot r^6} \
r &= x^2+y^2 \
\begin{bmatrix}
x\
y
\end{bmatrix} &= \begin{bmatrix}
X_c/Z_c \
Y_c/Z_c
\end{bmatrix} \
\end{align
}

Pinhole re-projection (3rd step)

In pinhole model incident angle and sensor matrix radius are connected with each other according to equation:

\begin{align}
r &= f \cdot \tan{\theta} \to r = \tan \theta, & \text{when focal length is normalized} \
\end{align}

Camera extrinsic parameters

Extrinsic parameters describe the camera's position and orientation in the 3D world. They consist of:

  • Rotation ($R$): The orientation of the camera relative to the world.
  • Translation ($T$): The position of the camera center relative to the world origin.

The extrinsic parameters form the 3x4 camera extrinsic matrix $[R|T]$, which transforms world coordinates into
camera coordinates. They are stored in SceneLayoutNode/SceneNode transformations.

Camera intrinsic parameters

Intrinsic parameters stores the properties of the camera itself that affect how 3D points are mapped to
2D coordinates. They include:

  • Focal length ($f$) - the distance from the pinhole to the image plane.
  • Principal point ($c_x, c_y$) - the point where the principal axis intersects the image plane (often the image
    center).

Intrinsic parameters are passed in the lens parameters. More about the influence of intrinsic parameters on ray
generation can be found in INTRO_Lens.py tutorial.

PinholeLens - render and undistortion

In our implementation of the PinholeLens, we can adjust parameters presented in the section above.
In this example, we will adjust camera focal length and principal point, as well as all distortions parameters.

PinholeLens setup and visualization

# Image resolution width = 1000 height = 1000 # Camera intrinsic parameters camera_fx = 750 camera_fy = 750 camera_cx = width / 2 camera_cy = height / 2 # Camera distortion parameters dist_k1 = -0.2 dist_k2 = 0.1660696363841201 dist_k3 = -0.1046838971475154 dist_k4 = -0.05589875277472606 dist_k5 = 0 dist_k6 = 0 dist_p1 = -0.00104636649317819 dist_p2 = -0.00168027284332261 dist_s1 = 0.01253 dist_s2 = 0.05434 dist_s3 = 0.08452 dist_s4 = 0.01137 dist_sampling_factor = 4 dist_range_extension = 0

We can pass the above parameters to the PinholeLens through parameter provider, and pass parametrized lens to
VisibleLightRenderStep as in SENSOR_PinholeLens tutorial.

from skyrenderer.render_chain import RenderChain, VisibleLightRenderStep, Denoiser from skyrenderer.render_chain.camera_steps.Lens.pinhole_lens import PinholeLens pinhole_params = PinholeLens.create_parameter_provider( renderer_context, fx=camera_fx, fy=camera_fy, cx_relative=0.5, cy_relative=0.5, dist_k1=dist_k1, dist_k2=dist_k2, dist_k3=dist_k3, dist_k4=dist_k4, dist_k5=dist_k5, dist_k6=dist_k6, dist_p1=dist_p1, dist_p2=dist_p2, dist_s1=dist_s1, dist_s2=dist_s2, dist_s3=dist_s3, dist_s4=dist_s4, dist_sampling_factor=dist_sampling_factor, dist_range_extension=dist_range_extension, ) lens = PinholeLens(renderer_context, parameter_provider=pinhole_params) rs = VisibleLightRenderStep( renderer_context, lens=lens, origin_name="camera_CAM_NUL", target_name="top_node", ) renderer_context.define_render_chain( RenderChain(render_steps=[rs, Denoiser(renderer_context)], width=width, height=height) ) distorted_image = scene_composer.get_render() scene_composer.visualize(distorted_image)
pinhole-lens-parameters_1_resourcesTutorial
2025-02-14 11:54:44,228 | skyrenderer.utils.time_measurement | INFO: Setup time: 2.97 seconds 2025-02-14 11:54:44,755 | skyrenderer.utils.time_measurement | INFO: Context update time: 526 ms 2025-02-14 11:54:45,472 | skyrenderer.utils.time_measurement | INFO: Key points calculation time: 0 ms 2025-02-14 11:54:45,472 | skyrenderer.utils.time_measurement | INFO: Render time: 716 ms

Image undistortion

Our Pinhole model implementation is compatible with the state-of-the-art solutions (e.g. OpenCV), therefore we can
perform undistortion on the rendered image with 3rd party library:

import cv2 as cv import numpy as np camera_matrix = np.array([[camera_fx, 0, camera_cx], [0, camera_fy, camera_cy], [0, 0, 1]], np.float) dist_coeffs = np.array( [dist_k1, dist_k2, dist_p1, dist_p2, dist_k3, dist_k4, dist_k5, dist_k6, dist_s1, dist_s2, dist_s3, dist_s4], np.float, ) img_undistorted = cv.undistort(distorted_image, camera_matrix, dist_coeffs) scene_composer.visualize(img_undistorted)
pinhole-lens-parameters_2_resourcesTutorial

PinholeLens - parameters change

fx, fy - focal length

width = 450 height = 450 renders = {} scene_composer.setup_pinhole_lens(width, height, fx=200) renders["fx=200"] = scene_composer.get_render() scene_composer.setup_pinhole_lens(width, height, fx=915) renders["fx=915(default)"] = scene_composer.get_render() scene_composer.setup_pinhole_lens(width, height, fx=1500) renders["fx=1500"] = scene_composer.get_render() scene_composer.setup_pinhole_lens(width, height, fx=450, fy=200) renders["fy=200"] = scene_composer.get_render() scene_composer.setup_pinhole_lens(width, height, fy=915) renders["fy=915(default)"] = scene_composer.get_render() scene_composer.setup_pinhole_lens(width, height, fy=1500) renders["fy=1500"] = scene_composer.get_render() scene_composer.setup_pinhole_lens(width, height, fx=200, fy=200) renders["fx=fy=200"] = scene_composer.get_render() scene_composer.setup_pinhole_lens(width, height, fx=915, fy=915) renders["fx=fy=915(default)"] = scene_composer.get_render() scene_composer.setup_pinhole_lens(width, height, fx=1500, fy=1500) renders["fx=fy=1500"] = scene_composer.get_render() scene_composer.visualize_grid_desc(renders, shape=(3 height, 3 width), n_cols=3, font_scale=2)
pinhole-lens-parameters_3_resourcesTutorial
2025-02-14 11:54:46,301 | skyrenderer.utils.time_measurement | INFO: Setup time: 180 ms 2025-02-14 11:54:46,785 | skyrenderer.utils.time_measurement | INFO: Context update time: 484 ms 2025-02-14 11:54:46,957 | skyrenderer.utils.time_measurement | INFO: Key points calculation time: 0 ms 2025-02-14 11:54:46,958 | skyrenderer.utils.time_measurement | INFO: Render time: 172 ms 2025-02-14 11:54:47,004 | skyrenderer.utils.time_measurement | INFO: Setup time: 43 ms 2025-02-14 11:54:47,483 | skyrenderer.utils.time_measurement | INFO: Context update time: 478 ms 2025-02-14 11:54:48,363 | skyrenderer.utils.time_measurement | INFO: Key points calculation time: 0 ms 2025-02-14 11:54:48,363 | skyrenderer.utils.time_measurement | INFO: Render time: 880 ms 2025-02-14 11:54:48,410 | skyrenderer.utils.time_measurement | INFO: Setup time: 42 ms 2025-02-14 11:54:48,877 | skyrenderer.utils.time_measurement | INFO: Context update time: 466 ms 2025-02-14 11:54:49,102 | skyrenderer.utils.time_measurement | INFO: Key points calculation time: 0 ms 2025-02-14 11:54:49,102 | skyrenderer.utils.time_measurement | INFO: Render time: 225 ms 2025-02-14 11:54:50,152 | skyrenderer.utils.time_measurement | INFO: Setup time: 1.05 seconds 2025-02-14 11:54:50,655 | skyrenderer.utils.time_measurement | INFO: Context update time: 502 ms 2025-02-14 11:54:51,519 | skyrenderer.utils.time_measurement | INFO: Key points calculation time: 0 ms 2025-02-14 11:54:51,520 | skyrenderer.utils.time_measurement | INFO: Render time: 865 ms 2025-02-14 11:54:51,577 | skyrenderer.utils.time_measurement | INFO: Setup time: 52 ms 2025-02-14 11:54:52,088 | skyrenderer.utils.time_measurement | INFO: Context update time: 510 ms 2025-02-14 11:54:52,264 | skyrenderer.utils.time_measurement | INFO: Key points calculation time: 0 ms 2025-02-14 11:54:52,264 | skyrenderer.utils.time_measurement | INFO: Render time: 176 ms 2025-02-14 11:54:52,312 | skyrenderer.utils.time_measurement | INFO: Setup time: 43 ms 2025-02-14 11:54:52,823 | skyrenderer.utils.time_measurement | INFO: Context update time: 511 ms 2025-02-14 11:54:54,005 | skyrenderer.utils.time_measurement | INFO: Key points calculation time: 0 ms 2025-02-14 11:54:54,005 | skyrenderer.utils.time_measurement | INFO: Render time: 1.18 seconds 2025-02-14 11:54:54,055 | skyrenderer.utils.time_measurement | INFO: Setup time: 45 ms 2025-02-14 11:54:54,523 | skyrenderer.utils.time_measurement | INFO: Context update time: 467 ms 2025-02-14 11:54:54,688 | skyrenderer.utils.time_measurement | INFO: Key points calculation time: 0 ms 2025-02-14 11:54:54,689 | skyrenderer.utils.time_measurement | INFO: Render time: 165 ms 2025-02-14 11:54:54,736 | skyrenderer.utils.time_measurement | INFO: Setup time: 43 ms 2025-02-14 11:54:55,239 | skyrenderer.utils.time_measurement | INFO: Context update time: 503 ms 2025-02-14 11:54:55,416 | skyrenderer.utils.time_measurement | INFO: Key points calculation time: 0 ms 2025-02-14 11:54:55,417 | skyrenderer.utils.time_measurement | INFO: Render time: 177 ms 2025-02-14 11:54:55,464 | skyrenderer.utils.time_measurement | INFO: Setup time: 43 ms 2025-02-14 11:54:55,951 | skyrenderer.utils.time_measurement | INFO: Context update time: 486 ms 2025-02-14 11:54:56,125 | skyrenderer.utils.time_measurement | INFO: Key points calculation time: 0 ms 2025-02-14 11:54:56,126 | skyrenderer.utils.time_measurement | INFO: Render time: 175 ms

cx_relative, cy_relative - center of projection

renders = {} scene_composer.setup_pinhole_lens(width, height, cx_relative=0.2) renders["cx=0.2"] = scene_composer.get_render() scene_composer.setup_pinhole_lens(width, height, cx_relative=0.5) renders["cx=0.5(default)"] = scene_composer.get_render() scene_composer.setup_pinhole_lens(width, height, cx_relative=0.8) renders["cx=0.8"] = scene_composer.get_render() scene_composer.setup_pinhole_lens(width, height, cx_relative=0.5, cy_relative=0.2) renders["cy=0.2"] = scene_composer.get_render() scene_composer.setup_pinhole_lens(width, height, cy_relative=0.5) renders["cy=0.5(default)"] = scene_composer.get_render() scene_composer.setup_pinhole_lens(width, height, cy_relative=0.8) renders["cy=0.8"] = scene_composer.get_render() scene_composer.setup_pinhole_lens(width, height, cx_relative=0.2, cy_relative=0.2) renders["cx=cy=0.2"] = scene_composer.get_render() scene_composer.setup_pinhole_lens(width, height, cx_relative=0.5, cy_relative=0.5) renders["cx=cy=0.5(default)"] = scene_composer.get_render() scene_composer.setup_pinhole_lens(width, height, cx_relative=0.8, cy_relative=0.8) renders["cx=cy=0.8"] = scene_composer.get_render() scene_composer.visualize_grid_desc(renders, shape=(3 height, 3 width), n_cols=3, font_scale=2)
pinhole-lens-parameters_4_resourcesTutorial
2025-02-14 11:54:56,763 | skyrenderer.utils.time_measurement | INFO: Setup time: 44 ms 2025-02-14 11:54:57,261 | skyrenderer.utils.time_measurement | INFO: Context update time: 497 ms 2025-02-14 11:54:58,202 | skyrenderer.utils.time_measurement | INFO: Key points calculation time: 0 ms 2025-02-14 11:54:58,203 | skyrenderer.utils.time_measurement | INFO: Render time: 941 ms 2025-02-14 11:54:58,251 | skyrenderer.utils.time_measurement | INFO: Setup time: 44 ms 2025-02-14 11:54:58,754 | skyrenderer.utils.time_measurement | INFO: Context update time: 503 ms 2025-02-14 11:54:59,782 | skyrenderer.utils.time_measurement | INFO: Key points calculation time: 0 ms 2025-02-14 11:54:59,783 | skyrenderer.utils.time_measurement | INFO: Render time: 1.03 seconds 2025-02-14 11:54:59,829 | skyrenderer.utils.time_measurement | INFO: Setup time: 43 ms 2025-02-14 11:55:00,387 | skyrenderer.utils.time_measurement | INFO: Context update time: 556 ms 2025-02-14 11:55:01,388 | skyrenderer.utils.time_measurement | INFO: Key points calculation time: 0 ms 2025-02-14 11:55:01,388 | skyrenderer.utils.time_measurement | INFO: Render time: 1.00 seconds 2025-02-14 11:55:01,436 | skyrenderer.utils.time_measurement | INFO: Setup time: 43 ms 2025-02-14 11:55:01,970 | skyrenderer.utils.time_measurement | INFO: Context update time: 533 ms 2025-02-14 11:55:02,992 | skyrenderer.utils.time_measurement | INFO: Key points calculation time: 0 ms 2025-02-14 11:55:02,992 | skyrenderer.utils.time_measurement | INFO: Render time: 1.02 seconds 2025-02-14 11:55:03,040 | skyrenderer.utils.time_measurement | INFO: Setup time: 43 ms 2025-02-14 11:55:03,559 | skyrenderer.utils.time_measurement | INFO: Context update time: 519 ms 2025-02-14 11:55:04,542 | skyrenderer.utils.time_measurement | INFO: Key points calculation time: 0 ms 2025-02-14 11:55:04,543 | skyrenderer.utils.time_measurement | INFO: Render time: 983 ms 2025-02-14 11:55:04,592 | skyrenderer.utils.time_measurement | INFO: Setup time: 45 ms 2025-02-14 11:55:05,087 | skyrenderer.utils.time_measurement | INFO: Context update time: 494 ms 2025-02-14 11:55:06,090 | skyrenderer.utils.time_measurement | INFO: Key points calculation time: 0 ms 2025-02-14 11:55:06,091 | skyrenderer.utils.time_measurement | INFO: Render time: 1.00 seconds 2025-02-14 11:55:06,138 | skyrenderer.utils.time_measurement | INFO: Setup time: 43 ms 2025-02-14 11:55:06,634 | skyrenderer.utils.time_measurement | INFO: Context update time: 496 ms 2025-02-14 11:55:07,603 | skyrenderer.utils.time_measurement | INFO: Key points calculation time: 0 ms 2025-02-14 11:55:07,604 | skyrenderer.utils.time_measurement | INFO: Render time: 969 ms 2025-02-14 11:55:07,651 | skyrenderer.utils.time_measurement | INFO: Setup time: 43 ms 2025-02-14 11:55:08,155 | skyrenderer.utils.time_measurement | INFO: Context update time: 503 ms 2025-02-14 11:55:09,200 | skyrenderer.utils.time_measurement | INFO: Key points calculation time: 0 ms 2025-02-14 11:55:09,201 | skyrenderer.utils.time_measurement | INFO: Render time: 1.05 seconds 2025-02-14 11:55:09,248 | skyrenderer.utils.time_measurement | INFO: Setup time: 43 ms 2025-02-14 11:55:09,737 | skyrenderer.utils.time_measurement | INFO: Context update time: 488 ms 2025-02-14 11:55:09,917 | skyrenderer.utils.time_measurement | INFO: Key points calculation time: 0 ms 2025-02-14 11:55:09,918 | skyrenderer.utils.time_measurement | INFO: Render time: 181 ms

aperture_jitter and focus_distance

apperture_jitter is the parameter responsible for aperture related to focal length. The higher value, the smaller
DOF of a camera. If 0, infinite DOF is present. On the other hand, focus_distance is parameter holding a value of
plane distance in meters. It is used, when aperture jitter is nonzero.

renders = {} scene_composer.setup_pinhole_lens(width, height, cx_relative=0.5, cy_relative=0.5) renders["aperture=0(default)"] = scene_composer.get_render() scene_composer.setup_pinhole_lens(width, height, aperture_jitter=1, focus_distance=5) renders["aperture=1,focus_dist=5"] = scene_composer.get_render() scene_composer.setup_pinhole_lens(width, height, aperture_jitter=1, focus_distance=15) renders["aperture=1,focus_dist=15"] = scene_composer.get_render() scene_composer.setup_pinhole_lens(width, height, aperture_jitter=1, focus_distance=25) renders["aperture=1,focus_dist=25"] = scene_composer.get_render() scene_composer.visualize_grid_desc( renders, shape=(height, 4 * width), n_cols=4, font_scale=1, text_position=(20, 20) )
pinhole-lens-parameters_5_resourcesTutorial
2025-02-14 11:55:10,666 | skyrenderer.utils.time_measurement | INFO: Setup time: 43 ms 2025-02-14 11:55:11,150 | skyrenderer.utils.time_measurement | INFO: Context update time: 483 ms 2025-02-14 11:55:11,327 | skyrenderer.utils.time_measurement | INFO: Key points calculation time: 0 ms 2025-02-14 11:55:11,328 | skyrenderer.utils.time_measurement | INFO: Render time: 177 ms 2025-02-14 11:55:11,376 | skyrenderer.utils.time_measurement | INFO: Setup time: 43 ms 2025-02-14 11:55:11,879 | skyrenderer.utils.time_measurement | INFO: Context update time: 502 ms 2025-02-14 11:55:12,894 | skyrenderer.utils.time_measurement | INFO: Key points calculation time: 0 ms 2025-02-14 11:55:12,895 | skyrenderer.utils.time_measurement | INFO: Render time: 1.02 seconds 2025-02-14 11:55:12,942 | skyrenderer.utils.time_measurement | INFO: Setup time: 42 ms 2025-02-14 11:55:13,453 | skyrenderer.utils.time_measurement | INFO: Context update time: 510 ms 2025-02-14 11:55:14,461 | skyrenderer.utils.time_measurement | INFO: Key points calculation time: 0 ms 2025-02-14 11:55:14,461 | skyrenderer.utils.time_measurement | INFO: Render time: 1.01 seconds 2025-02-14 11:55:14,508 | skyrenderer.utils.time_measurement | INFO: Setup time: 43 ms 2025-02-14 11:55:15,006 | skyrenderer.utils.time_measurement | INFO: Context update time: 497 ms 2025-02-14 11:55:15,182 | skyrenderer.utils.time_measurement | INFO: Key points calculation time: 0 ms 2025-02-14 11:55:15,183 | skyrenderer.utils.time_measurement | INFO: Render time: 176 ms

Summary

In this section you have learnt:

  • All parameters from pinhole lens mathematical theory can be set in PinholeLens's create_parameter_provider
    method.
  • Pinhole lens model follows generic projection model described in INTRO_Lens.
  • Model-based parameters of pinhole lens are compatible with popular CV libraries like OpenCV, and renders can be
    undistorted using 3rd party library.