For determining a state of a pavement from surroundings sensor data, local data received from at least one device that measures a local pavement state or coefficient of friction is merged with camera image data received from a camera for imaging a pavement extending in front of the vehicle. When analyzing the camera image data, the local data representing the measured pavement state or coefficient of friction is assigned to individual image sectors of a camera image while taking odometric and time information into account to achieve proper correspondence, and the local data is taken into account for supporting and/or plausibility-checking of an anticipatory estimation of the local coefficient of friction or pavement state based on the camera image data.
Legal claims defining the scope of protection. Each claim is shown in both the original legal language and a plain English translation.
1. A method of determining a state of a pavement on which a vehicle is driving, comprising merging locally measured data received from at least one locally measuring device that measures a local pavement state or coefficient of friction of the pavement, with camera data of a camera image received from a camera, and performing an image analysis comprising analyzing the camera data, which involves: sub-dividing the camera image into a two-dimensional grid of grid cells representing image sectors of the camera image in a plane of the pavement, assigning the locally measured data representing the local pavement state or coefficient of friction respectively to individual ones of the image sectors of the camera image in the camera data while taking odometric information of the vehicle and time information into account, and taking the locally measured data representing the local pavement state or coefficient of friction into account for support and/or plausibilization of an anticipatory and locally resolved coefficient-of-friction estimation or state-of-pavement determination based on the camera data.
A method for determining the pavement state while driving involves combining data from a local sensor (measuring pavement state or friction) with camera images. The camera image is divided into a grid representing pavement sectors. The sensor data is matched to specific image sectors, considering vehicle movement and time. This sensor data supports or validates estimates of pavement friction or state derived from analyzing the camera images. This provides a way to use visual data and direct measurements to understand road conditions.
2. The method according to claim 1 , wherein the image analysis includes assigning the locally measured data representing the local pavement state or coefficient of friction to at least one pavement segment respectively represented in at least one of the image sectors in the camera image when the odometric information and the time information reveal that the locally measured data has been measured respectively at the at least one pavement segment.
In the pavement state determination method, the sensor data (pavement state or friction) is assigned to specific pavement segments visible in the camera image. This assignment happens when vehicle movement and time information confirm that the sensor measured the pavement at that exact location shown in the image. This ensures precise matching between sensor readings and visual representations of the road.
3. The method according to claim 1 , wherein the image analysis provides a classification of the individual image sectors in the camera image based on particular features of pavement segments depicted in the image sectors.
In the pavement state determination method, the camera image analysis classifies each grid cell (image sector) based on features of the pavement segment seen in that cell. For example, visual textures, color, and detected objects may be used to determine the type and condition of the road surface in that specific area of the image. This image analysis informs understanding of the pavement.
4. The method according to claim 1 , wherein a total number of the grid cells of the grid is determined dependent on a homogeneity of the pavement.
In the pavement state determination method, the total number of grid cells used to divide the camera image depends on how uniform the pavement is. If the pavement is relatively homogeneous, fewer grid cells are needed. If the pavement condition varies significantly, a finer grid (more cells) is used for greater detail. This enables dynamic grid size adjustment.
5. The method according to claim 1 , wherein a total number of the grid cells of the grid is determined dependent on a current driving situation and/or a criticality thereof.
In the pavement state determination method, the total number of grid cells used to divide the camera image depends on the current driving situation or its criticality. A more critical driving situation (e.g., emergency braking) might require a finer grid (more cells) for more detailed pavement analysis. Normal driving can utilize a coarser grid. This prioritizes critical situations.
6. The method according to claim 1 , wherein a total number of the grid cells of the grid is determined dependent on an available computing power for performing the method.
In the pavement state determination method, the total number of grid cells used to divide the camera image depends on available computing power. More grid cells require more processing. The number of grid cells is dynamically adapted to balance detail and computational cost. A powerful processor allows for a finer grid.
7. The method according to claim 1 , wherein a result of the image analysis of the camera data is predictively applied afterwards, whilst taking the local pavement state or coefficient of friction assigned to the camera image into account, to a subsequently acquired camera image.
In the pavement state determination method, the image analysis result is used to predict pavement state in subsequent camera images. The system takes the locally measured pavement state or coefficient of friction assigned to the *current* camera image into account when analyzing the *next* camera image. This creates a predictive, iterative process for assessing road conditions.
8. The method according to claim 1 , further comprising calculating a vehicle corridor from a predicted movement trajectory of the vehicle, by which vehicle corridor, positions of the at least one locally measuring device comprising individual wheels of the vehicle and/or at least one locally measuring sensor are predictively assigned to pavement segments respectively represented in the image sectors of the camera image, said pavement segments extending in front of the vehicle.
The pavement state determination method calculates a vehicle corridor based on the predicted path of the vehicle. The system then predictively assigns the positions of the local sensor(s) (e.g., wheel sensors) to pavement segments visible in the camera image that are in front of the vehicle and within this corridor. This allows the system to anticipate and correlate sensor readings with visual data along the vehicle's path.
9. The method according to claim 1 , further comprising assigning probability values to a respective one of the image sectors or a respective pavement segment represented in the respective image sector, said probability values indicating with what probability the respective image sector or the respective pavement segment is to be assigned to a first class and to at least a second class.
In the pavement state determination method, probability values are assigned to each image sector (grid cell) or the pavement segment it represents. These probabilities indicate how likely the sector or segment belongs to different pavement classes (e.g., "dry," "wet," "icy"). This enables probabilistic assessment of pavement conditions, reflecting uncertainty and potential mixed conditions.
10. The method according to claim 1 , wherein a mono camera is used as the camera.
In the pavement state determination method, a single (mono) camera is used to capture the camera images.
11. The method according to claim 1 , wherein a stereo camera is used as the camera.
In the pavement state determination method, a stereo camera is used to capture the camera images, enabling depth perception and three-dimensional analysis of the pavement.
12. The method according to claim 1 , wherein a radar sensor, an ultrasonic sensor or an optical sensor is used as the locally measuring device, which is configured for locally determining a three-dimensional shape of a pavement surface of the pavement.
In the pavement state determination method, the local sensor is a radar, ultrasonic, or optical sensor that measures the three-dimensional shape of the pavement surface. This sensor provides detailed geometric information about the road.
13. The method according to claim 1 , wherein at least one measuring device that measures and/or derives a local coefficient of friction from speed signals of a vehicle wheel is used as the locally measuring device.
In the pavement state determination method, the local sensor measures or derives the pavement friction from wheel speed signals. This utilizes existing vehicle data to estimate road slipperiness.
14. A device for determining a state of a pavement on which a vehicle is driving, comprising a camera that provides camera data of a camera image, at least one locally measuring device configured to measure a local pavement state or coefficient of friction of the pavement and to provide corresponding locally measured data, a merger device configured to merge the locally measured data with the camera data, and an image analysis device configured to perform an image analysis comprising analyzing the camera data, wherein, the image analysis device is further configured to sub-divide the camera image into a two-dimensional grid of grid cells representing image sectors of the camera image in a plane of the pavement, and to assign the locally measured data representing the local pavement state or coefficient of friction respectively to individual ones of the image sectors of the camera image in the camera data while taking odometric information of the vehicle and time information into account, and to take the locally measured data representing the local pavement state or coefficient of friction into account for support and/or plausibilization of an anticipatory and locally resolved coefficient-of-friction estimation or state-of-pavement determination based on the camera data.
A device for determining pavement state includes a camera, a local sensor (measuring pavement state or friction), a merger, and an image analyzer. The camera provides images, and the sensor provides local measurements. The merger combines these data sources. The image analyzer divides the camera image into a grid, assigns sensor data to grid cells (accounting for vehicle movement and time), and uses this data to support or validate estimates of pavement friction or state derived from the camera images.
15. A method of determining a surface condition of a driving surface on which a vehicle is driving, comprising steps: a) with a camera on said vehicle, producing camera data including a camera image of a selected surface area of said driving surface ahead in front of said vehicle; b) performing an image analysis and classification of said selected surface area in said camera image to determine, among predetermined classes, an estimated class of a surface condition comprising an estimated pavement state or an estimated coefficient of friction of said selected surface area; c) driving said vehicle forward whereby said selected surface area comes into a sensing range of a locally measuring sensor on said vehicle, and using time information and odometric information of said vehicle to achieve sensing registration of said locally measuring sensor with said selected surface area; d) with said locally measuring sensor, sensing locally measured data representing an actual surface condition comprising an actual pavement state or an actual coefficient of friction of said selected surface area; e) merging said locally measured data representing said actual surface condition with said estimated class of said surface condition for said selected surface area, and thereby assigning a new value of said actual surface condition to said estimated class of said surface condition, or validating, plausibilizing or correcting a previously assigned value of said actual surface condition that had previously been assigned to said estimated class of said surface condition; and f) repeating said steps a) to e), with regard to a subsequent selected surface area of said driving surface ahead in front of said vehicle.
A method for determining driving surface condition involves: (a) Capturing camera images of the road ahead. (b) Analyzing the image to classify the surface condition (pavement state/friction). (c) Driving forward, tracking vehicle movement and time to register sensor readings with the previously viewed area. (d) Using a local sensor to measure the actual surface condition (pavement state/friction). (e) Combining sensor data with the initial image-based classification to refine the estimated surface condition, validating or correcting the initial assessment. (f) Repeating the process for subsequent road areas.
Cooperative Patent Classification codes for this invention. Click any code to explore related patents in that topic.
December 9, 2013
June 13, 2017
Browse 5M+ US patents with plain-English claim translations and AI-generated analysis.