An apparatus includes: a generating unit which generates, from each of images respectively obtained at viewpoint positions, one of different-viewpoint images which corresponds to an image at a virtual viewpoint position different from the viewpoint positions, the different-viewpoint image including a hole area in which a pixel value is missing; a calculating unit which calculates, for each of processing units respectively in predetermined areas in the different-viewpoint images, a hole density indicating, with respect to the predetermined area, a ratio of one of the hole areas in the processing units in the different-viewpoint images; a calculating unit which calculates, for each processing unit, a combination ratio of the different-viewpoint image, based on the hole density of the processing unit co-located with an other one of the processing units in an other one of the different-viewpoint images; and a combining unit which combines the different-viewpoint images, based on the combination ratios.
Legal claims defining the scope of protection. Each claim is shown in both the original legal language and a plain English translation.
1. A different-viewpoint image generating apparatus comprising: a processor; and a non-transitory computer-readable medium having stored thereon executable instructions that, when executed by the processor, cause the different-viewpoint image generating apparatus to function as: a different-viewpoint image generating unit configured to generate, from each of two or more images respectively obtained at two or more viewpoint positions, one of two or more different-viewpoint images which correspond to an image obtainable at a virtual viewpoint position different from the two or more viewpoint positions, based on distance information indicating a depth of a pixel in one of the two or more images, the one of the two or more different-viewpoint images including a hole area in which a pixel value is missing; a hole density calculating unit configured to calculate, for each of processing units respectively in predetermined areas in the two or more different-viewpoint images, a hole density indicating, with respect to the predetermined area, a ratio of one of the hole areas in the processing units in the two or more different-viewpoint images, the processing unit being made up of one or more pixels; a combination ratio calculating unit configured to calculate, for each of the processing units, a combination ratio of one of the two or more different-viewpoint images, based on the hole density of the processing unit which is co-located with an other one of the processing units in an other one of the two or more different-viewpoint images; and a different-viewpoint image combining unit configured to combine the two or more different-viewpoint images, based on the combination ratios of the processing units, wherein the hole density calculating unit is configured to calculate, as a plurality of the hole densities of the co-located processing units in the two or more different-viewpoint images, a plurality of the ratios of the hole areas in windows which are the predetermined areas having the co-located processing units as centers of the predetermined areas, and wherein the hole density calculating unit is configured to calculate a plurality of the hole densities by assigning different weights to a hole area located at a central part of the window and a hole area located at a peripheral part of the window.
A system generates new views of a scene from multiple input images taken from different positions. It calculates depth information for pixels in the input images, and then generates new images from a virtual camera position, which often results in missing pixel data ("holes"). The system divides each new image into processing units (areas made up of one or more pixels). It calculates "hole density" for each processing unit, representing the ratio of missing pixels in that area. It then calculates a combination ratio for each image based on the hole densities of corresponding processing units in the other images. Finally, it combines the different images based on these ratios, giving more weight to the images that have fewer holes in a specific area. The system calculates hole density using a window centered on each processing unit, and the calculation weights holes in the center of the window differently than holes on the periphery.
2. The different-viewpoint image generating apparatus according to claim 1 , wherein the executable instructions, when executed by the processor, cause the different-viewpoint image generating apparatus to further function as a hole area interpolating unit configured to interpolate each of the hole areas in the two or more different-viewpoint images, using pixel values within a corresponding one of the two or more different-viewpoint images, wherein the different-viewpoint image combining unit is configured to combine the two or more different-viewpoint images including the interpolated hole areas, based on the combination ratios.
The system as described above, which generates new views of a scene from multiple input images, calculates depth information, generates new images with holes, calculates hole densities, combination ratios, and then combines images based on these ratios, also includes a hole filling component. This component interpolates the missing pixel values within each of the generated images *before* the images are combined, using existing pixel values in the same image to fill in the missing data. The final image combination uses these filled-in images.
3. The different-viewpoint image generating apparatus according to claim 1 , wherein the executable instructions, when executed by the processor, cause the different-viewpoint image generating apparatus to further function as a hole area interpolating unit configured to interpolate the hole areas within images combined by the different-viewpoint image combining unit, using pixel values within the images.
The system as described above, which generates new views of a scene from multiple input images, calculates depth information, generates new images with holes, calculates hole densities, combination ratios, and then combines images based on these ratios, also includes a hole filling component. This component interpolates the missing pixel values *after* the images have been combined, using existing pixel values in the combined image to fill in the remaining holes.
4. The different-viewpoint image generating apparatus according to claim 1 , wherein the combination ratio calculating unit is configured to calculate, for the co-located processing units in the two or more different-viewpoint images, a plurality of the combination ratios which become larger as a plurality of the hole densities of the co-located processing units become smaller.
In the system described above, which generates new views of a scene from multiple input images, calculates depth information, generates new images with holes, calculates hole densities, combination ratios, and then combines images based on these ratios, the combination ratio is inversely proportional to the hole density. That is, for corresponding areas in different images, the combination ratio for an image becomes *larger* (meaning it contributes more to the final combined image) as the hole density in that area becomes *smaller*.
5. The different-viewpoint image generating apparatus according to claim 1 , wherein, when hole densities of co-located processing units are equal to each other between the two or more different-viewpoint images, the combination ratio calculating unit is configured to calculate, for the co-located processing units having the equal hole densities, combination ratios which become larger as the positions of the two or more different-viewpoint images become closer to the position of the virtual viewpoint position.
In the system described above, which generates new views of a scene from multiple input images, calculates depth information, generates new images with holes, calculates hole densities, combination ratios, and then combines images based on these ratios, if corresponding areas in different images have *equal* hole densities, the combination ratios are calculated based on the proximity of the input images to the virtual viewpoint. The images that were captured *closer* to the virtual viewpoint will have *larger* combination ratios and contribute more to the final image.
6. The different-viewpoint image generating apparatus according to claim 1 , wherein the processing unit is a pixel, the hole density calculating unit is configured to calculate a hole density for each of pixels co-located in the two or more different-viewpoint images, and the combination ratio calculating unit is configured to calculate, for each of the co-located pixels, the combination ratio of one of the two or more different-viewpoint images.
In the system described above, which generates new views of a scene from multiple input images, calculates depth information, generates new images with holes, calculates hole densities, combination ratios, and then combines images based on these ratios, the "processing unit" is a single pixel. Therefore, the system calculates a hole density for each pixel location across all images and then calculates the combination ratio for each pixel location in each image, to determine how much each input pixel contributes to the final output pixel.
7. A different-viewpoint image generating method comprising: generating, from each of two or more images respectively obtained at two or more viewpoint positions, one of two or more different-viewpoint images which correspond to an image obtainable at a virtual viewpoint position different from the two or more viewpoint positions, based on distance information indicating a depth of a pixel in one of the two or more images, the one of the two or more different-viewpoint images including a hole area in which a pixel value is missing; calculating, for each of processing units respectively in predetermined areas in the two or more different-viewpoint images, a hole density indicating, with respect to the predetermined area, a ratio of one of the hole areas in the processing units in the two or more different-viewpoint images, the processing unit being made up of one or more pixels; calculating, for each of the processing units, a combination ratio of one of the two or more different-viewpoint images, based on the hole density of the processing unit which is co-located with an other one of the processing units in an other one of the two or more different-viewpoint images; and combining the two or more different-viewpoint images, based on the combination ratios of the processing units, wherein the calculating of the hole density comprises calculating, as a plurality of the hole densities of the co-located processing units in the two or more different-viewpoint images, a plurality of the ratios of the hole areas in windows which are the predetermined areas having the co-located processing units as centers of the predetermined areas, and wherein, in the calculating of the hole density, a plurality of the hole densities are calculated by assigning different weights to a hole area located at a central part of the window and a hole area located at a peripheral part of the window.
A method generates new views of a scene from multiple input images taken from different positions. It calculates depth information for pixels in the input images, and then generates new images from a virtual camera position, which often results in missing pixel data ("holes"). The method divides each new image into processing units (areas made up of one or more pixels). It calculates "hole density" for each processing unit, representing the ratio of missing pixels in that area. It then calculates a combination ratio for each image based on the hole densities of corresponding processing units in the other images. Finally, it combines the different images based on these ratios, giving more weight to the images that have fewer holes in a specific area. The method calculates hole density using a window centered on each processing unit, and the calculation weights holes in the center of the window differently than holes on the periphery.
8. A non-transitory computer-readable recording medium for use in a computer, the recording medium having a computer program recorded thereon for causing the computer to execute the different-viewpoint image generating method according to claim 7 .
A computer-readable storage medium stores a program. When executed, the program performs a method that generates new views of a scene from multiple input images taken from different positions. It calculates depth information for pixels in the input images, and then generates new images from a virtual camera position, which often results in missing pixel data ("holes"). The method divides each new image into processing units (areas made up of one or more pixels). It calculates "hole density" for each processing unit, representing the ratio of missing pixels in that area. It then calculates a combination ratio for each image based on the hole densities of corresponding processing units in the other images. Finally, it combines the different images based on these ratios, giving more weight to the images that have fewer holes in a specific area. The method calculates hole density using a window centered on each processing unit, and the calculation weights holes in the center of the window differently than holes on the periphery.
9. An integrated circuit comprising: a different-viewpoint image generating circuit configured to generate, from each of two or more images respectively obtained at two or more viewpoint positions, one of two or more different-viewpoint images which correspond to an image obtainable at a virtual viewpoint position different from the two or more viewpoint positions, based on distance information indicating a depth of a pixel in one of the two or more images, the one of the two or more different-viewpoint images including a hole area in which a pixel value is missing; a hole density calculating circuit configured to calculate, for each of processing units respectively in predetermined areas in the two or more different-viewpoint images, a hole density indicating, with respect to the predetermined area, a ratio of one of the hole areas in the processing units in the two or more different-viewpoint images, the processing unit being made up of one or more pixels; a combination ratio calculating circuit configured to calculate, for each of the processing units, a combination ratio of one of the two or more different-viewpoint images, based on the hole density of the processing unit which is co-located with an other one of the processing units in an other one of the two or more different-viewpoint images; and a different-viewpoint image combining circuit configured to combine the two or more different-viewpoint images, based on the combination ratios of the processing units, wherein the hole density calculating circuit is configured to calculate, as a plurality of the hole densities of the co-located processing units in the two or more different-viewpoint images, a plurality of the ratios of the hole areas in windows which are the predetermined areas having the co-located processing units as centers of the predetermined areas, and wherein the hole density calculating circuit is configured to calculate a plurality of the hole densities by assigning different weights to a hole area located at a central part of the window and a hole area located at a peripheral part of the window.
An integrated circuit generates new views of a scene from multiple input images taken from different positions. It calculates depth information for pixels in the input images, and then generates new images from a virtual camera position, which often results in missing pixel data ("holes"). The system divides each new image into processing units (areas made up of one or more pixels). It calculates "hole density" for each processing unit, representing the ratio of missing pixels in that area. It then calculates a combination ratio for each image based on the hole densities of corresponding processing units in the other images. Finally, it combines the different images based on these ratios, giving more weight to the images that have fewer holes in a specific area. The system calculates hole density using a window centered on each processing unit, and the calculation weights holes in the center of the window differently than holes on the periphery.
10. The different-viewpoint image generating apparatus according to claim 1 , wherein the hole density calculating unit is configured to calculate the plurality of the hole densities by assigning a larger weight to the hole area located at the central part of the window than the hole area located at the peripheral part of the window.
In the system described above, which generates new views of a scene from multiple input images, calculates depth information, generates new images with holes, calculates hole densities, combination ratios, and then combines images based on these ratios, when the hole density is calculated using a window, the hole area at the center of the window is weighted more heavily than the hole area at the periphery of the window. This emphasizes the importance of having valid pixels closer to the processing unit's center.
Cooperative Patent Classification codes for this invention. Click any code to explore related patents in that topic.
October 16, 2013
March 14, 2017
Browse 5M+ US patents with plain-English claim translations and AI-generated analysis.