Methods and systems for distributing remote assistance to facilitate robotic object manipulation are provided herein. Regions of a model of objects in an environment of a robotic manipulator may be determined, where each region corresponds to a different subset of objects with which the robotic manipulator is configured to perform a respective task. Certain tasks may be identified, and a priority queue of requests for remote assistance associated with the identified tasks may be determined based on expected times at which the robotic manipulator will perform the identified tasks. At least one remote assistor device may then be requested, according to the priority queue, to provide remote assistance with the identified tasks. The robotic manipulator may then be caused to perform the identified tasks based on responses to the requesting, received from the at least one remote assistor device, that indicate how to perform the identified tasks.
Legal claims defining the scope of protection. Each claim is shown in both the original legal language and a plain English translation.
1. A system, comprising: a remote assistor device; and a control system for a robotic device, wherein the control system is configured to: identify a first task and a second task, wherein the first task involves a first object and the second task involves a second object; determine a first expected time at which the robotic device is expected to perform the first task; determine a second expected time at which the robotic device is expected to perform the second task; based on the first expected time and the second expected time, determine an order in which to transmit a first request for remote assistance with the first task and a second request for remote assistance with the second task; transmit the first request and the second request to the remote assistor device in accordance with the order; receive, from the remote assistor device, a first response including data relating to performance of the first task; receive, from the remote assistor device, a second response including data relating to performance of the second task; and cause the robotic device to perform both the first task and the second task based on the first response and the second response.
A robotic system includes a robot control system and a remote assistance device. The control system identifies tasks for the robot involving different objects (first task with object A, second with object B). It predicts when each task will occur and prioritizes requests for remote assistance based on these times. The system sends these prioritized requests to the remote assistant, receives data on how to perform each task from the assistant and then commands the robot to execute both tasks according to received assistance. This allows efficient and timely remote guidance for robotic operations.
2. The system of claim 1 , wherein the control system is further configured to: determine a first distance of the first object from the robotic device; determine a second distance of the second object from the robotic device; and determine the order in which to transmit the first request and the second request based on the first distance and the second distance.
In the robotic system with remote assistance, described previously, the robot control system considers the distance of each object from the robot when prioritizing requests for help. The system determines the distance of object A (first task) and object B (second task) from the robotic device. Shorter distances (closer objects) may receive higher priority, or vice versa, depending on the system configuration and task requirements. The order in which assistance requests are sent to the remote assistant is based both on the time the task is expected to occur and the proximity of the object involved.
3. The system of claim 1 , wherein the control system is further configured to determine the order in which to transmit the first request and the second request based on whether at least one of the first object and the second object is a fragile object.
In the robotic system with remote assistance, described previously, the robot control system considers whether the object is fragile when prioritizing requests for help. The system checks if object A (first task) or object B (second task) are fragile. Tasks involving fragile objects may receive higher priority, requesting assistance sooner. The order in which assistance requests are sent to the remote assistant is based on whether either object requires extra care in handling.
4. The system of claim 1 , wherein one or both of the first task and the second task involves identifying a virtual boundary line representing at least a portion of a border of the respective object involved in the respective task.
In the robotic system with remote assistance, described previously, one or both of the robot's tasks may involve identifying a virtual boundary line representing at least a portion of the border of the object involved in the task. The robotic system is enabled to determine where the edge of an object is by means of a virtual line that serves as a portion of the object's border. The remote assistant may be required to assist in this determination or provide guidance on how to handle objects whose boundaries are identified in this manner.
5. The system of claim 1 , wherein the remote assistor device is configured to: receive, from the control system, the first and second request in the order; receive a first input relating to performance of the first task; receive a second input relating to performance of the second task; based on the first input, determine the data relating to performance of the first task; based on the second input, determine the data relating to performance of the second task; transmit, to the control system, the first response including the data relating to performance of the first task; and transmit, to the control system, the second response including the data relating to performance of the second task.
In the robotic system, the remote assistance device receives prioritized task requests from the robot control system. A human operator inputs data relating to performance of the first task and performance of the second task. The remote assistor device then processes this information and determines the best way to perform each task based on the input. It then transmits responses containing this task performance data back to the robot control system, enabling the robot to execute the tasks with remote guidance. The first and second requests are received in the determined order.
6. The system of claim 5 , wherein the remote assistor device includes a display, and wherein the remote assistor device is further configured to: based on the first request, cause the display to include first information relating to performance of the first task; and based on the second request, cause the display to include second information relating to performance of the second task.
In the robotic system with remote assistance, the remote assistant's device includes a display screen. When the device receives a request for assistance, the display shows information about the related task. For example, when a request about manipulating object A is received, the screen would show video or relevant data to the remote user. This allows the human operator to visualize the robotic task and provide better assistance based on the displayed information.
7. The system of claim 1 , wherein the control system is further configured to: determine a first confidence score indicating a level of confidence in performance of the first task; determine a second confidence score indicating a level of confidence in performance of the second task; and identify the first task and the second task based on the first confidence score and the second confidence score.
In the robotic system, the robot control system determines a confidence score for each task, indicating its certainty in successfully completing the task without help. The system uses these confidence scores when deciding whether to request remote assistance. If the confidence score for a task involving object A is low, a remote assistance request is made earlier and more urgently than if the score is high. This allows the remote assistant to be engaged when the robot is most likely to need guidance.
8. The system of claim 1 , wherein the control system is further configured to: before determining the order in which to transmit the first request and the second request: determine a remote assistor rating that indicates, for the remote assistor device, a history of whether the robotic device has correctly performed one or more previous tasks based on one or more responses the robotic device has previously received from the remote assistor device; make a determination that the remote assistor rating exceeds a predetermined threshold rating; and responsive to the determination being that the remote assistor rating exceeds the predetermined threshold rating, select the remote assistor device for transmission of the first request and the second request.
In the robotic system, before sending a request to a remote assistor, the control system checks a "remote assistor rating" for that device. This rating reflects the assistor's past performance. If the rating is above a certain threshold (meaning the assistor has been helpful in the past), the system proceeds to send the request for assistance. If the rating is low, the system might choose a different remote assistor or rely on other methods. The robot utilizes only remote assistants who are known to be effective, to improve success rates.
9. The system of claim 1 , wherein the control system is further configured to: determine task performance statistics relating to performance of the first task and the second task by the robotic device based on the first response and the second response; and transmit the task performance statistics to the remote assistor device.
In the robotic system with remote assistance, the robot control system gathers statistics on how well the robot performs tasks, based on the responses from remote assistors. For example, it tracks the time it takes to complete a task with assistance, the accuracy of the manipulation, or the number of errors made. This performance data is then sent back to the remote assistor device, giving the human operator feedback on the impact of their assistance and helping them improve their guidance in the future.
10. The system of claim 1 , further comprising: a sensor system configured to acquire video data; and a robotic manipulator configured to manipulate an object, wherein the first task involves the robotic manipulator manipulating the first object, wherein the second task involves the robotic manipulator manipulating the second object, wherein the control system is further configured to: cause the sensor system to acquire a first video feed of performance of the first task; cause the sensor system to acquire a second video feed of performance of the second task; transmit the first video feed and the second video feed to the remote assistor device.
The robotic system includes a sensor system (e.g., cameras) that capture video. A robotic manipulator arm is used to physically manipulate objects. When the robot performs a task (e.g., moving object A), the sensor system records a video feed of the process. This video is sent to the remote assistor device, allowing the human operator to see what the robot is doing and provide real-time feedback. Separate video feeds are used for different tasks (object A and object B), so the assistant can view the relevant information for each.
11. A method performed by a control system of a robotic device, the method comprising: identifying a first task and a second task, wherein the first task involves a first object and the second task involves a second object; determining a first expected time at which the robotic device is expected to perform the first task; determining a second expected time at which the robotic device is expected to perform the second task; based on the first expected time and the second expected time, determining an order in which to transmit a first request for remote assistance with the first task and a second request for remote assistance with the second task; transmitting the first request and the second request to a remote assistor device in accordance with the order; receiving, from the remote assistor device, a first response including data relating to performance of the first task; receiving, from the remote assistor device, a second response including data relating to performance of the second task; and causing the robotic device to perform both the first task and the second task based on the first response and the second response.
A robot control system performs the following steps: it identifies two tasks for a robot (first with object A, second with object B). It then predicts when each task will need to be done. Based on these times, it determines the order in which to request remote assistance. It sends assistance requests to a remote device and receives information about how to perform each task. Finally, it instructs the robot to execute the tasks based on the remote assistance.
12. The method of claim 11 , further comprising: determining a first distance of the first object from the robotic device; determining a second distance of the second object from the robotic device; and determining the order in which to transmit the first request and the second request based on the first distance and the second distance.
In the method performed by a robotic control system described previously, the control system determines the distance of the first object (involved in the first task) and the second object (involved in the second task) from the robotic device. The order in which the requests for assistance are transmitted to the remote assistor is determined based on the distance of the two objects from the robotic device.
13. The method of claim 11 , further comprising: determining the order in which to transmit the first request and the second request based on whether at least one of the first object and the second object is a fragile object.
In the method performed by a robotic control system described previously, the control system determines if at least one of the objects (involved in the tasks) are fragile. The order in which the requests for assistance are transmitted to the remote assistor is determined based on whether either of the objects are fragile. The method prioritizes assistance with fragile objects.
14. The method of claim 11 , wherein one or both of the first task and the second task involves (i) identifying a virtual boundary line separating two adjacent objects, the two adjacent objects including the respective object involved in the respective task, and (ii) manipulating the respective object based on the identified virtual boundary line.
In the method performed by a robotic control system described previously, at least one task involves identifying a "virtual boundary line" separating adjacent objects. The object is then manipulated based on this boundary line. This virtual boundary line aids the robot in distinguishing between objects. Assistance from a remote operator may involve verifying or correcting this identified virtual boundary.
15. The method of claim 11 , wherein one or both of the first task and the second task involves (i) identifying a location on the respective object involved in the respective task and (ii) gripping the respective object at the location with a gripper of a robotic manipulator.
In the method performed by a robotic control system described previously, at least one of the tasks involves (i) identifying a location on the object and (ii) gripping the object at the identified location with a gripper. The location is identified on the object by a remote operator using remote assistance.
16. The method of claim 11 , wherein one or both of the first task and the second task involves determining a sequenced order in which a robotic manipulator is expected to manipulate a respective set of objects, the respective set of objects including the respective object involved in the respective task.
In the method performed by a robotic control system described previously, at least one of the tasks involves determining a sequenced order in which the robotic manipulator is expected to manipulate a set of objects, including the object involved in the task. The remote assistor device is used to determine the best sequence.
17. A non-transitory computer readable medium having stored thereon instructions that, upon execution by a control system for a robotic device, cause the control system to perform operations comprising: identifying a first task and a second task, wherein the first task involves a first object and the second task involves a second object; determining a first expected time at which the robotic device is expected to perform the first task; determining a second expected time at which the robotic device is expected to perform the second task; based on the first expected time and the second expected time, determining an order in which to transmit a first request for remote assistance with the first task and a second request for remote assistance with the second task; transmitting the first request and the second request to a remote assistor device in accordance with the order; receiving, from the remote assistor device, a first response including data relating to performance of the first task; receiving, from the remote assistor device, a second response including data relating to performance of the second task; and causing the robotic device to perform both the first task and the second task based on the first response and the second response.
A non-transitory computer-readable medium (like a flash drive) contains instructions that, when executed by a robot control system, cause the system to: identify two tasks for a robot (first with object A, second with object B), predict when each task will occur, prioritize assistance requests based on these times, send the requests to a remote assistant, receive data on how to perform each task, and instruct the robot to execute the tasks based on the remote assistance.
18. The non-transitory computer readable medium of claim 17 , wherein one or both of the first task and the second task involves (i) identifying a virtual boundary line separating two adjacent objects, the two adjacent objects including the respective object involved in the respective task, and (ii) manipulating the respective object based on the identified virtual boundary line.
The non-transitory computer readable medium from the previous description has the instructions that the robot's tasks involve identifying a "virtual boundary line" separating adjacent objects. The object is then manipulated based on this boundary line. The virtual boundary line is identified and used in the determination of how to handle the object.
19. The non-transitory computer readable medium of claim 17 , wherein the order is a first order the operations further comprising: based on the first expected time and the second expected time, determining a second order in which the robotic device is expected to cause the robotic device to perform both the first task and the second task; and causing the robotic device to perform both the first task and the second task in the second order.
The non-transitory computer readable medium from the previous description has instructions such that the robot determines a first order in which to transmit the requests for assistance. It also determines a second order in which the robot will perform the tasks. The robot then performs both the first task and the second task based on the second order.
20. The non-transitory computer readable medium of claim 19 , the operations further comprising: determining a first distance of the first object from the robotic device; determining a second distance of the second object from the robotic device; and determining the second order in which the robotic device is expected to cause the robotic device to perform both the first task and the second task based on the first distance and the second distance.
The non-transitory computer readable medium from the previous description has instructions such that the robot determines the distance of the first object and the second object from the robotic device. The second order is determined based on the distances of the objects from the robotic device. The second order is the order in which the robot performs the tasks.
Cooperative Patent Classification codes for this invention. Click any code to explore related patents in that topic.
October 5, 2016
May 16, 2017
Browse 5M+ US patents with plain-English claim translations and AI-generated analysis.