Automated vehicle repair estimation by aggregate ensembling of multiple artificial intelligence functions

This application is a reissue of U.S. Pat. No. 11,556,902, issued on Jan. 17, 2023.

The present application claims priority to U.S. Provisional Patent Application No. 62/908,348, filed Sep. 30, 2019, entitled “Methods For Processing Data From an Artificial Intelligence Model And Devices Thereof”; U.S. Provisional Patent Application No. 62/908,354, filed Sep. 30, 2019, entitled “Methods For Processing Data From an Artificial Intelligence Model Using Voting Technique And Devices Thereof”; and U.S. Provisional Patent Application No. 62/908,361, filed Sep. 30, 2019, entitled “Methods For Processing Data From an Artificial Intelligence Model Using Preferential Technique and Devices Thereof”; the disclosures thereof incorporated by reference herein initstheirentirety.

The present application is related to U.S. patent application Ser. No. 17/039,262, filed Sep. 30, 2020, now U.S. Pat. No. 11,823,137, entitled “Automated Vehicle Repair Estimation by Voting Ensembling of Multiple Artificial Intelligence Functions”; U.S. patent application Ser. No. 17/039,287, filed Sep. 30, 2020, now U.S. Pat. No. 11,836,684, entitled “Automated Vehicle Repair Estimation by Preferential Ensembling of Multiple Artificial Intelligence Functions”; U.S. patent application Ser. No. 17/039,311, filed Sep. 30, 2020, now U.S. Pat. No. 11,887,063, entitled “Automated Vehicle Repair Estimation by Random Ensembling of Multiple Artificial Intelligence Functions”; and U.S. patent application Ser. No. 17/039,339, filed Sep. 30, 2020, now U.S. Pat. No. 11,797,952, entitled “Automated Vehicle Repair Estimation by Adaptive Ensembling of Multiple Artificial Intelligence Functions.”

The disclosed technology relates generally to artificial intelligence (AI), and more particularly some embodiments relate to the use of in vehicle repair estimation.

A claimed solution rooted in computer technology overcomes problems specifically arising in the realm of computer technology.

In general, one aspect disclosed features a system, comprising: a hardware processor; and a non-transitory machine-readable storage medium encoded with instructions executable by the hardware processor to perform a method comprising: receiving a plurality of vehicle repair recommendation sets for a damaged vehicle, wherein each of the vehicle repair recommendation sets identifies at least one recommended vehicle repair operation of a plurality of the vehicle repair operations for the damaged vehicle; aggregating a plurality of the recommended vehicle repair operations; generating a composite vehicle repair recommendation set that identifies the aggregated recommended vehicle repair operations; and providing the composite vehicle repair recommendation set to one or more vehicle repair insurance claims management systems.

Embodiments of the system may include one or more of the following features. In some embodiments, each of the vehicle repair recommendation sets identifies a plurality of images of the damaged vehicle; and the method further comprises: selecting one or more of the images of the damaged vehicle, and identifying the selected one or more of the images in the generated composite vehicle repair recommendation set. In some embodiments, each of the vehicle repair recommendation sets identifies a score and/or confidence percentage for the recommended vehicle repair operation; and the method further comprises: identifying the scores for the recommended vehicle repair operations in the generated composite vehicle repair recommendation set. In some embodiments, each of the vehicle repair recommendation sets includes one or more images of the damaged vehicle; selecting a plurality of the recommended vehicle repair operations comprises: providing the vehicle repair recommendation sets to an artificial intelligence function; and the artificial intelligence function is trained using a plurality of vehicle repair training sets, wherein each vehicle repair training set comprises: one or more images of a further damaged vehicle, and a composite vehicle repair recommendation set for the further damaged vehicle. In some embodiments, each of the vehicle repair recommendation sets is generated by a respective further artificial intelligence function. In some embodiments, each of the further artificial intelligence functions is trained. In some embodiments, the method further comprises requesting one or more of the further artificial intelligence functions be re-trained when a predetermined event occurs. In some embodiments, aggregating a plurality of the recommended vehicle repair operations comprises: aggregating the plurality of the recommended vehicle repair operations based on statistical aggregation methodologies comprising at least one of mean, max, min, variance, and standard deviation.

In general, one aspect disclosed features a non-transitory machine-readable storage medium encoded with instructions executable by a hardware processor of a computing component, the machine-readable storage medium comprising instructions to cause the hardware processor to perform a method comprising: receiving a plurality of vehicle repair recommendation sets for a damaged vehicle, wherein each of the vehicle repair recommendation sets identifies at least one recommended vehicle repair operation of a plurality of the vehicle repair operations for the damaged vehicle; aggregating a plurality of the recommended vehicle repair operations; generating a composite vehicle repair recommendation set that identifies the aggregated recommended vehicle repair operations; and providing the composite vehicle repair recommendation set to one or more vehicle repair insurance claims management systems.

Embodiments of the non-transitory machine-readable storage medium may include one or more of the following features. In some embodiments, each of the vehicle repair recommendation sets identifies a plurality of images of the damaged vehicle; and the method further comprises: selecting one or more of the images of the damaged vehicle, and identifying the selected one or more of the images in the generated composite vehicle repair recommendation set. In some embodiments, each of the vehicle repair recommendation sets identifies a score and/or confidence percentage for the recommended vehicle repair operation; and the method further comprises: identifying the scores for the recommended vehicle repair operations in the generated composite vehicle repair recommendation set. In some embodiments, each of the vehicle repair recommendation sets includes one or more images of the damaged vehicle; selecting a plurality of the recommended vehicle repair operations comprises: providing the vehicle repair recommendation sets to an artificial intelligence function; and the artificial intelligence function is trained using a plurality of vehicle repair training sets, wherein each vehicle repair training set comprises: one or more images of a further damaged vehicle, and a composite vehicle repair recommendation set for the further damaged vehicle. In some embodiments, each of the vehicle repair recommendation sets is generated by a respective further artificial intelligence function. In some embodiments, each of the further artificial intelligence functions is trained. In some embodiments, the method further comprises requesting one or more of the further artificial intelligence functions be re-trained when a predetermined event occurs. In some embodiments, aggregating a plurality of the recommended vehicle repair operations comprises: aggregating the plurality of the recommended vehicle repair operations based on statistical aggregation methodologies comprising at least one of mean, max, min, variance, and standard deviation.

In general, one aspect disclosed features a method comprising: receiving a plurality of vehicle repair recommendation sets for a damaged vehicle, wherein each of the vehicle repair recommendation sets identifies at least one recommended vehicle repair operation of a plurality of the vehicle repair operations for the damaged vehicle; aggregating a plurality of the recommended vehicle repair operations; generating a composite vehicle repair recommendation set that identifies the aggregated recommended vehicle repair operations; and providing the composite vehicle repair recommendation set to one or more vehicle repair insurance claims management systems.

Embodiments of the method may include one or more of the following features. In some embodiments, each of the vehicle repair recommendation sets identifies a plurality of images of the damaged vehicle; and the method further comprises: selecting one or more of the images of the damaged vehicle, and identifying the selected one or more of the images in the generated composite vehicle repair recommendation set. In some embodiments, each of the vehicle repair recommendation sets identifies a score and/or confidence percentage for the recommended vehicle repair operation; and the method further comprises: identifying the scores for the recommended vehicle repair operations in the generated composite vehicle repair recommendation set. In some embodiments, each of the vehicle repair recommendation sets includes one or more images of the damaged vehicle; selecting a plurality of the recommended vehicle repair operations comprises: providing the vehicle repair recommendation sets to an artificial intelligence function; and the artificial intelligence function is trained using a plurality of vehicle repair training sets, wherein each vehicle repair training set comprises: one or more images of a further damaged vehicle, and a composite vehicle repair recommendation set for the further damaged vehicle. In some embodiments, each of the vehicle repair recommendation sets is generated by a respective further artificial intelligence function. In some embodiments, each of the further artificial intelligence functions is trained. In some embodiments, the method further comprises requesting one or more of the further artificial intelligence functions be re-trained when a predetermined event occurs. In some embodiments, aggregating a plurality of the recommended vehicle repair operations comprises: aggregating the plurality of the recommended vehicle repair operations based on statistical aggregation methodologies comprising at least one of mean, max, min, variance, and standard deviation.

The figures are not exhaustive and do not limit the present disclosure to the precise form disclosed.

With the advent of high-power, cost effective computing systems came the increased automation of numerous facets of our contemporary society. In the insurance and other casualty and loss industries, for example, computerized claims estimating, processing, tracking and payment systems have long been in use to streamline processes and to expedite claims handling and closure.

Advances in Artificial Intelligence (AI) have enabled the use of AI to assist in the estimating process. However, AI implementations may differ markedly. For example, given the same input, two different AI functions may produce quite different estimates. Embodiments of the disclosed technology exploit these variations in AI implementations to harness the power of multiple AI functions in generating a vehicle repair estimate. In various embodiments, a claim package is provided to a plurality of AI functions in various ways, and the estimates output by the AI functions are combined in various ways.

Some embodiments employ an aggregating approach. According to these embodiments, the same claim package is provided to each of the AI functions, and the estimates produced by the AI functions are aggregated.

Some embodiments employ a voting approach. According to these embodiments, the same claim package is provided to each of the AI functions, and the estimates produced by the AI functions are combined according to scores assigned to the estimates by the AI functions.

Some embodiments employ a preferential approach. According to these embodiments, the same claim package is provided to each of the AI functions, and each of the AI functions is assigned a respective rank. The estimates produced by the AI functions are selected according to the ranks.

Some embodiments employ a random approach. According to these embodiments, different portions of the claim package are provided to the AI functions in a random manner, and the estimates produced by the AI functions are combined.

Some embodiments employ an adaptive approach. According to these embodiments, weights are learned for each operation for each component for each AI function. The same claim package is provided to each of the AI functions, and the estimates produced by the AI functions are selected according to the learned weights.

In this description, various embodiments are disclosed for vehicle repair estimate workflow automation. However, embodiments of the disclosed technology apply to other processes as well. For example, embodiments may apply to generating estimates for medical procedures, and the like. These and other applications will be apparent to one skilled in the relevant art after reading this description. Before describing embodiments of the disclosed technology in detail, it is useful to describe an example environment in which the disclosed technology may be implemented.

illustrates a vehicle estimate automation systemaccording to some embodiments of the disclosed technology. Multiple users may be involved in the vehicle repair workflow. For example, referring to, the users may include the insured, a claims adjuster, a repairersuch as an employee of a repair shop, an independent appraiser, and the like. Each user may employ a respective client device,,,. Each client device may be implemented as a desktop computer, laptop computer, smart phone, smart glasses, embedded computers and displays, diagnostic devices and the like.

The systemmay include a vehicle repair estimate automation tool, which may be implemented as one or more software packages executing on one or more server computers. Each user may employ a respective client device,,,to access the toolover a networksuch as the Internet.

The toolmay include a plurality of AI functions. The AI functionsmay be implemented in any manner. For example, one or more of the AI functionsmay be implemented as trained machine learning models. The machine learning models may include computer vision machine learning models, natural language processing machine learning models, and the like. The systemmay include one or more databases, which may store vehicle repair procedures, completed estimates, estimates in process, data regarding parts, part costs, labor, labor costs, and the like. The databasesmay include one or more natural language processing (NLP) techniques. The natural language processing databases may include rules, documents, and the like for use with the natural language processing machine learning models. In some embodiments, a NLP machine learning model may be trained with vehicle repair content. The vehicle repair content may include vehicle specifications, vehicle repair procedures, position statements, parts catalogs, and the like. The NLP machine learning model may ingest and process the vehicle repair content to generate machine learning rules and processed NLP content databases. The NLP machine learning model may further curate the ingested vehicle repair content through other artificial intelligence technologies including image analysis, text mining, deep analysis, and the like.

The systemmay include a claims management system. The claims management systemmay be operated, for example, by a vehicle insurer.

illustrates a processfor vehicle repair workflow according to some embodiments of the disclosed technology. The elements of the disclosed processes are presented in a particular order. However, it should be understood that, in various embodiments, one or more elements may be performed in a different order, in parallel, or omitted. Referring to, the processmay begin with a vehicle accident, at, and may continue with the vehicle owner reporting the accident to an insurance company, and taking the vehicle to a repair facility, at. Alternatively, the owner may take the vehicle to a repair facility, at, which may report the accident to the insurance company, at.

Next, a vehicle damage assessment is performed, at. For example, a staff appraiser of an insurance company may visit the damaged vehicle to take photos of the damage. Alternatively, the owner may send photos of the damaged vehicle to the insurance company. Next, the process may include the generation of a vehicle repair estimate, at, as described below in detail. Based on the vehicle repair estimate, the repair of the vehicle may take place, at. When the repair is complete, the repaired vehicle may be delivered to the vehicle owner, at.

illustrates a vehicle repair estimate automation systemaccording to some embodiments of the disclosed technology. Elements of the systemmay be implemented within the vehicle repair estimating systemof. Referring to, the systemmay include ensembling AI functionand a plurality of evaluating AI functionsA, B, C. While for clarity of explanation only three evaluating AI functionsare shown and discussed, it should be appreciated that any number of evaluating AI functionsmay be used.

In some embodiments, one or more of the evaluating AI functionsmay be implemented and/or operated by the same entity that implements and/or operates ensembling AI function. In some embodiments, one or more of the evaluating AI functionsmay be implemented and/or operated by an entity that is different from the entity that implements and/or operates ensembling AI function. For example, one or more of the evaluating AI functionsmay be implemented and/or operated by a third-party vendor or the like. One advantage of this arrangement is that evaluating AI functions may be added and removed as desired.

As noted above, the AI functions,may be implemented in any manner. The systemmay include an applications programming interface (API) gateway (GW)to provide an interface between the ensembling AI functionand other systems. In the example of, one or more claims management systemsmay access the ensembling AI functionthrough the API GW.

The claims management systemmay submit a claim packageto the ensembling AI function. The claim package may include data describing a damaged vehicle. The data may include images such as photos of the damaged vehicle, video of the damaged vehicle, audio recordings describing the damaged vehicle, text describing the damaged vehicle, and the like.

The ensembling AI functionmay distribute the claim packageto the evaluating AI functions. Referring to, the ensembling AI functionmay distribute claim packagesA,B,C to the evaluating AI functionsA,B,C, respectively. In some embodiments, the claim packagesA,B,C may be the same. In other embodiments, the claim packagesA,B,C may differ. For example, each of the claim packagesA,B,C may be a different subset of the claim package.

Referring to, the evaluating AI functionsA,B,C may generate recommendation setsA,B,C, respectively. The evaluating AI functionsmay be implemented in different manners, so that when given the same claim package, the evaluating AI functionsmay generate different recommendation sets.

A recommendation setmay identify at least one recommended vehicle repair operation, of a plurality of the vehicle repair operations, for the damaged vehicle. For example, a recommended vehicle repair operation may indicate that a front bumper of the vehicle should be replaced. A recommendation setmay identify a score for a recommended vehicle repair operation. Each score may be a floating-point value that indicates a projected accuracy of the recommended vehicle repair operation, with higher floating-point values indicating greater projected accuracy. A recommendation setmay include metadata objects of the damaged vehicle, for example including one or more of the images, in the corresponding claim package. In some instances, in addition to scores, a mathematical and/or statistical confidence percentage may also be included to indicate confidence around AI inferred accuracy of each recommended operation. Higher confidence percentages for an AI decision may indicate higher confidence in the AI decision.

The ensembling AI functionmay generate a composite recommendation setbased on the recommendation setsgenerated by the evaluating AI functions, as described in detail below. The composite recommendation setmay identify at least one recommended vehicle repair operation, a score for a recommended vehicle repair operation, one or more of the images in the corresponding claim package, and the like. The ensembling AI functionmay provide the composite recommendation setto one or more claims management systems.

illustrates a vehicle repair estimate automation processaccording to some embodiments of the disclosed technology. The processmay be performed, for example, by the vehicle repair estimating systemsandof. Referring to, the processmay include receiving a claim package, at. In the example of, the ensembling AI functionmay receive a claim packagefrom one or more claims management systemsthrough the API GW.

Referring again to, the processmay include determining a configuration of the vehicle repair estimate automation process, at. In some embodiments, the processmay be configured to employ one or more ensembling techniques. In the example of, the ensembling techniques may include aggregate ensembling, at, voting ensembling, at, preferential ensembling, at, adaptive ensembling, at, and random ensembling, at. In the example of, the ensembling techniques may be performed by the ensembling AI functionin conjunction with the evaluating AI functions. Each of these techniques is described in detail below.

Referring again to, the processmay include providing the resulting composite recommendation set, at. In the example of, the ensembling AI functionmay provide the composite recommendation setto one or more claims management systemsthrough the API GW.

illustrates an aggregate ensembling processaccording to some embodiments of the disclosed technology. The processmay be performed, for example, by the vehicle repair estimating systemsandof. Referring to, the processmay include distributing the claim package to the evaluating AI functions, at. In the example of, ensembling AI functionmay distribute claim packagesA,B,C to the evaluating AI functionsA,B,C, respectively.

Referring again to, the processmay include receiving a vehicle repair recommendation set from the evaluating AI functions, wherein each of the vehicle repair recommendation sets identifies at least one recommended vehicle repair operation of a plurality of the vehicle repair operations, at. In the example of, ensembling AI functionmay receive recommendation setsA,B,C from the evaluating AI functionsA,B,C, respectively.

Referring again to, the processmay include aggregating a plurality of the recommended vehicle repair operations, at. In the example of, the ensembling AI functionmay aggregate the received recommendation sets. In some embodiments, aggregating the received recommendation setsmay include combining two or more of the received recommendation sets. In some embodiments, aggregating may include aggregating the plurality of the recommended vehicle repair operations based on statistical aggregation methodologies comprising at least one of mean, max, min, variance, and standard deviation. In some embodiments, redundant recommended vehicle repair operations may be omitted from the resulting combination. Other aggregation techniques are contemplated as well.

Referring again to, the processmay include generating a composite vehicle repair recommendation set that identifies the aggregated recommended vehicle repair operations, at. In the example of, the ensembling AI functionmay aggregate the received recommendation sets to form the composite vehicle repair recommendation set.

Referring again to, the processmay include providing the composite vehicle repair recommendation set to one or more vehicle repair insurance claims management systems, at. In the example of, the ensembling AI functionmay provide the composite recommendation setto one or more claims management systems.

In some embodiments, one or more of the evaluating AI functionsmay include one or more trained AI functions. For example, an AI function may be implemented as a trained computer vision machine learning model. The machine learning model may be trained, for example, with images of other damaged vehicles and the corresponding vehicle repair operations applied to repair those vehicles. In these embodiments, a trained AI function may be retrained occasionally, for example when a predetermined retraining trigger event occurs. A retraining trigger event may be a function a defined lapsed period of time (say, every six months, for example), a result of a comparison between a pre-defined evaluation metric of an AI function and a sample hold out (also known as test data set), and may consider factors such as measuring precision, recall and F1 score (accuracy). When an evaluation metric falls below a certain predefined metric, retraining can be automatically retriggered prior to a lapse of a predefined period. For example, the ensembling AI functionmay compare the components of the recommendation sets received from an evaluating AI function against predetermined benchmarks, and based on the comparison, may declare a retraining trigger event.

Referring again to, the processmay include determining whether a predetermined retraining trigger event has occurred, at. The processmay include requesting one or more of the evaluating artificial intelligence functions be re-trained when a predetermined event occurs, at. In the example of, the ensembling AI functionmay determine whether a predetermined retraining trigger event has occurred, and may request one or more of the evaluating artificial intelligence functionsbe re-trained when the predetermined event occurs.

illustrates an example operationaccording to the aggregate ensembling processof. Referring to, the ensembling AI functionreceives a claim package from a claims management systemvia an API GW. For clarity, only the images in the claim package are discussed. However, it should be appreciated that the claim package may contain other data objects in addition to, or instead of, the images, and that those data objects may be treated in a similar manner.

The claim package example illustrated includes five images,,,,of a damaged vehicle. The ensembling AI functiondistributes the images to the evaluating AI functions. In this example, the ensembling AI functiondistributes all of the images to each of the evaluating AI functions. The evaluating AI functionsprocess the images.

Each of the evaluating AI functionsreturns a different recommendation set. The evaluating AI functionA returns a recommendation set that includes images,and two recommended operations with components and scores. The legend inindicates identifiers for components and operations. One of the recommended operations is LFB, OP,., which indicates replacement of the Lower Front Bumper with a score of 0.7. The other recommended operation is UFB, OP,., which indicates repair of the Upper Front Bumper with a score of 0.5. But while the examples presented herein describe AI function inferencing outcome examples such as component, labor operation, associated score, and the like, it should be understood that additional AI outcomes may include other factors, for example such as repair labor hours, type of damage, area of damage, and the like.

The evaluating AI functionB returns a recommendation set that includes images,and one recommended operation with component and score. The recommended operation is LF, OP,., which indicates repair of the Left Fender with a score of 0.6.

The evaluating AI functionC returns a recommendation set that includes images,,and two recommended operations with components and scores. One of the recommended operations is FB, OP,., which indicates repair of the front bumper with a score of 0.5. The other recommended operation is H, OP,., which indicates painless dent repair of the hood with a score of 0.4.

The ensembling AI functionmay aggregate these recommendation sets to form the composite vehicle repair recommendation set. In this example, the composite recommendation set includes a union of the images in the recommendation sets provided by the evaluating AI functions, namely images,,,. The composite recommendation set also includes a union of the recommended operations and scores in the recommendation sets provided by the evaluating AI functions.