Techniques are provided for client and server integration for scalable replication. A replication client transmits change records to a database server over a stream. The database server determines at least one batch comprising change records for at least one transaction. The database server generates dependency data for at least one change record in a batch based on at least one constraint identifier for at least one column. The database server determines an ordered grouping of the change records based on an operation type of each change record and the dependency data of each change record, wherein change records sharing operation types are grouped together unless a division based on the dependency data is determined. The database server generates a reordered transaction comprising a plurality of reordered operations based on the ordered grouping of the change records of the particular batch.
Legal claims defining the scope of protection. Each claim is shown in both the original legal language and a plain English translation.
1. A method comprising: receiving, by a replication client, change records for transactions executed on a source database by a source database server; when replication between the replication client and a target database server is in asynchronous mode: transmitting, by the replication client, a first plurality of change records to a target database server for replication on a target database; replicating, by the target database server, one or more first transactions, described by one or more change records in the first plurality of change records, without acknowledgement to the replication client when the one or more first transactions are committed; and detecting a synchronization condition; in response to detecting the synchronization condition, switching from asynchronous mode to synchronized mode by bringing the target database to a synchronized state corresponding to a state of the target database after a particular transaction has been committed; in synchronized mode: submitting, by the replication client, one or more database commands comprising one or more second transactions described by a second plurality of change records to the target database server; and executing, by the target database server, the one or more database commands with acknowledgement to the replication client when the one or more second transactions are committed; after acknowledgement of committing the one or more second transactions is received by the replication client, resuming replication in asynchronous mode; wherein the method is performed by one or more computing devices.
A system replicates database changes between a source and target database. A client sends database transaction changes to a server asynchronously. The server replicates these transactions without immediately acknowledging them. When a synchronization condition is detected (e.g., an unsupported transaction), the client switches to synchronized mode, bringing the target database to a consistent state. In synchronized mode, the client sends database commands, and the server acknowledges each committed transaction. After acknowledgement, the system returns to asynchronous replication. This allows scalable replication with on-demand synchronization, performed by one or more computing devices.
2. The method of claim 1 , further comprising: determining, by the target database server, at least one batch, each batch of the at least one batch comprising change records received from the replication client; for a particular batch selected from the at least one batch, the target database server: generating dependency data for at least one change record in the particular batch based on at least one constraint identifier for at least one column associated with the at least one change record; determining an ordered grouping of the change records of the particular batch based on an operation type of each change record of the particular batch and the dependency data, wherein change records of the particular batch that share operation types are grouped together unless a division based on the dependency data is determined; generating a reordered transaction comprising a plurality of reordered operations based on the ordered grouping of the change records of the particular batch; executing the reordered transaction; wherein the dependency data for the at least one change record comprises at least one hash value, wherein each selected hash value of the at least one hash value is generated by hashing a selected value of a selected change record with a selected constraint identifier of a selected column associated with the selected value; wherein the division based on the dependency data is determined when at least one hash value is shared.
In the database replication system from the previous description, the target database server groups received change records into batches. For each batch, it generates dependency data (hash values) for change records based on constraint identifiers of related columns. It then orders the change records based on operation type, grouping records with the same type together, unless dependency data indicates otherwise (shared hash values). A reordered transaction is created based on this grouping and executed. The dependency data uses hash values generated by hashing values from change records with constraint identifiers of the associated columns. A division is created when these hash values are shared across the change records.
3. The method of claim 1 , wherein the target database server comprises a plurality of apply processes, the method further comprising: assigning, by the target database server, the one or more first transactions to the plurality of apply processes; wherein each particular apply process of the plurality of apply processes is configured to asynchronously apply and commit the transactions of the one or more first transactions without acknowledgement to the replication client; wherein the plurality of apply processes is configured to process the one or more first transactions in parallel.
In the database replication system from a previous description, the target database server uses multiple "apply processes" to handle incoming transactions. Each apply process asynchronously applies and commits its assigned transactions without acknowledgement to the client. The apply processes operate in parallel, improving replication throughput. The one or more first transactions are assigned to these processes for faster replication performance.
4. The method of claim 1 : wherein detecting the synchronization condition comprises detecting, by the replication client, one or more change records, corresponding to the one or more second transactions, for which asynchronous replication is not supported; wherein bringing the target database to a synchronized state comprises transmitting, by the replication client, a synchronization command to the target database server; wherein executing, by the target database server, the one or more database commands is performed after receiving the synchronization command.
In the database replication system from a previous description, detecting the synchronization condition involves the client identifying change records for which asynchronous replication is not supported, such as schema changes. To switch to synchronized mode, the client sends a synchronization command to the server. The server then executes subsequent database commands only after receiving this synchronization command. This ensures that specific operations are handled in a synchronized manner.
5. The method of claim 1 , further comprising: detecting, by the target database server, that an unhandled event has occurred; determining that the unhandled event is an asynchronous event that is manageable without synchronizing the replication client and the target database server; storing, by the target database server, unhandled event data, wherein the unhandled event data comprises event occurrence data and event detail data.
In the database replication system from a previous description, the target database server detects unhandled events. If an unhandled event is an asynchronous event that doesn't require synchronization, the server stores data about the event, including occurrence and detail information. This allows the server to track and manage asynchronous events without interrupting the replication process.
6. The method of claim 5 , further comprising: periodically querying, by the replication client, the target database server for new event occurrence data in the unhandled event data; when new event occurrence data is detected indicating one or more new unhandled events, the replication client: querying the target database server for new event detail data corresponding to the new unhandled events; performing an event handling procedure based on the new event detail data.
In the system from the previous description, the replication client periodically checks the target database server for new unhandled events. When new events are detected, the client requests detailed information about them and performs an event-handling procedure based on this detail data. This allows the client to react to asynchronous events without constant synchronization.
7. The method of claim 6 , wherein the event handling procedure comprises changing a read position of the replication client, wherein the read position corresponds to a position in the plurality of change records.
In the system from the previous description, the event handling procedure involves changing the client's read position. The read position corresponds to a location in the stream of change records. Adjusting the read position allows the client to skip over or replay change records based on the detected event, ensuring data consistency.
8. One or more non-transitory computer-readable media storing one or more sequences of instructions that, when executed by one or more processors, cause: receiving change records for transactions executed on a source database by a source database server; when replication with a target database server is in asynchronous mode: transmitting a first plurality of change records to a target database server for replication on a target database; wherein one or more first transactions, described by one or more change records in the first plurality of change records, are replicated by the target database server without acknowledgement when the one or more first transactions are committed; and detecting a synchronization condition; in response to detecting the synchronization condition, switching from asynchronous to synchronized mode by transmitting a synchronization command to the target database server, wherein the synchronization command causes the target database server to bring the target database to a synchronized state corresponding to a state of the target database after a particular transaction has been committed; in synchronized mode, submitting one or more database commands comprising one or more second transactions to the target database server, wherein the one or more database commands cause the target database server to execute the one or more database commands, wherein executing the one or more database commands comprises the target database server committing the one or more second transactions with acknowledgement of the commit; receiving acknowledgement of committing the one or more second transactions; after receiving acknowledgement of committing the one or more second transactions, resuming replication in asynchronous mode.
A computer-readable medium stores instructions for replicating database changes. A client sends transaction changes to a server asynchronously. The server replicates transactions without immediate acknowledgement. When a synchronization condition is detected, the system switches to synchronized mode, sending a synchronization command to the target database, bringing it to a consistent state. In synchronized mode, the client sends database commands, and the server acknowledges each committed transaction. After acknowledgement, the system returns to asynchronous replication.
9. The one or more computer-readable media of claim 8 , wherein the one or more sequences of instructions include instructions that, when executed by the one or more processors, cause: periodically querying the target database server for new event occurrence data in unhandled event data maintained by the target database server, wherein the target database server maintains the unhandled event data asynchronously without notification for at least one event type; when new event occurrence data is detected indicating one or more new unhandled events, querying the target database server for new event detail data corresponding to the new unhandled events; performing an event handling procedure based on the new event detail data.
The computer-readable medium in the previous description also contains instructions to periodically query the target database server for new unhandled event data. When new events are detected, the client requests detailed information about them and performs an event-handling procedure. The target database server maintains unhandled event data asynchronously without notification for at least one event type.
10. The one or more computer-readable media of claim 9 , wherein the event handling procedure comprises changing a read position that corresponds to a position in the plurality of change records for transmitting.
In the computer-readable medium from the previous description, the event handling procedure involves changing the client's read position, which corresponds to a location in the stream of change records to be transmitted.
11. One or more non-transitory computer-readable media storing one or more sequences of instructions that, when executed by one or more processors, cause: receiving, change records from a replication client for transactions executed on a source database by a source database server; replicating the transactions on a target database; when replication with the replication client is in asynchronous mode: replicating one or more first transactions, described by a first plurality of change records received from the replication client, without acknowledgement to the replication client when the one or more first transactions are committed; receiving a synchronization command from the replication client; in response to receiving the synchronization command, switching to synchronized mode by bringing the target database to a synchronized state corresponding to a state of the target database after a particular transaction has been committed; notifying the replication client that the target database is in a synchronized state; in synchronized mode: receiving, from the replication client, one or more database commands comprising one or more second transactions; executing the one or more database commands; transmitting an acknowledgement to the replication client indicating that the one or more second transactions were committed; resuming replication with the replication client in asynchronous mode.
A computer-readable medium stores instructions for a target database server that replicates database changes from a client. In asynchronous mode, the server replicates transactions without acknowledging them. Upon receiving a synchronization command from the client, the server switches to synchronized mode, bringing the database to a consistent state and notifying the client. In synchronized mode, the server receives and executes database commands, then sends an acknowledgement to the client. Finally, replication resumes in asynchronous mode.
12. The one or more computer-readable media of claim 11 , wherein the one or more sequences of instructions include instructions that, when executed by the one or more processors, cause: determining at least one batch, each batch of the at least one batch comprising change records received from the replication client; for a particular batch selected from the at least one batch: generating dependency data for at least one change record in the particular batch based on at least one constraint identifier for at least one column associated with the at least one change record; determining an ordered grouping of the change records of the particular batch based on an operation type of each change record of the particular batch and the dependency data, wherein change records of the particular batch that share operation types are grouped together unless a division based on the dependency data is determined; generating a reordered transaction comprising a plurality of reordered operations based on the ordered grouping of the change records of the particular batch; executing the reordered transaction; wherein the dependency data for the at least one change record comprises at least one hash value, wherein each selected hash value of the at least one hash value is generated by hashing a selected value of a selected change record with a selected constraint identifier of a selected column associated with the selected value; wherein the division based on the dependency data is determined when at least one hash value is shared.
The computer-readable medium from the previous description includes instructions for the server to group change records into batches. For each batch, it generates dependency data (hash values) for change records based on constraint identifiers of related columns. It then orders the change records based on operation type, grouping records with the same type together, unless dependency data indicates otherwise (shared hash values). A reordered transaction is created and executed. The dependency data uses hash values generated by hashing values from change records with constraint identifiers of associated columns. A division is created when these hash values are shared across the change records.
13. The one or more computer-readable media of claim 11 , wherein the one or more sequences of instructions include instructions that cause: assigning the one or more first transactions to a plurality of apply processes; wherein each particular apply process of the plurality of apply processes is configured to asynchronously apply and commit the transactions of the one or more first transactions without acknowledgement to the replication client; wherein the plurality of apply processes is configured to process the one or more first transactions in parallel.
The computer-readable medium from a previous description includes instructions for the server to assign incoming transactions to multiple "apply processes." Each apply process asynchronously applies and commits its assigned transactions without acknowledgement to the client, and all the apply processes operate in parallel.
14. The one or more computer-readable media of claim 11 , wherein bringing the target database to a synchronized state corresponding to a state of the target database after a particular transaction has been committed comprises, after receiving the synchronization command, completing execution of transactions of the one or more first transactions up to the particular transaction; transmitting a notification to the replication client that the target database is in the synchronized state.
In the computer-readable medium from a previous description, bringing the target database to a synchronized state involves completing the execution of transactions up to a specific transaction after receiving the synchronization command, and then notifying the client that the database is synchronized.
15. The one or more computer-readable media of claim 11 , wherein the one or more sequences of instructions include instructions that, when executed by the one or more processors, cause: detecting that an unhandled event has occurred; storing unhandled event data corresponding to the unhandled event, wherein the unhandled event data comprises event occurrence data and event detail data.
The computer-readable medium from the previous description also includes instructions for detecting unhandled events and storing unhandled event data, including event occurrence and detail information, to be used later.
16. An apparatus comprising: a network interface that is coupled to a data network for receiving one or more packet flows therefrom; at least one processor; one or more stored sequences of instructions which, when executed by the at least one processor, cause the at least one processor to perform: receiving, over an asynchronous stream, a plurality of change records from a replication client over a stream of change records for a plurality of transactions executed on a source database by a source database server; replicating the plurality of transactions on a target database; when replication with the replication client is in asynchronous mode, asynchronously replicating one or more first transactions, described by change records in the plurality of change records, without acknowledgement when transactions of the one or more first transactions are committed; receiving a synchronization command from the replication client when replication is in asynchronous mode; in response to receiving the synchronization command, switching to synchronized mode by bringing the target database to a synchronized state corresponding to a state of the target database after a particular transaction has been committed; notifying the replication client that the target database is in a synchronized state; in synchronized mode, receiving, from the replication client, one or more database commands comprising one or more second transactions; executing the one or more database commands, wherein executing the one or more database commands comprises committing the one or more second transactions; transmitting an acknowledgement to the replication client indicating that the one or more second transactions were committed; resuming replication with the replication client in asynchronous node.
An apparatus includes a network interface and a processor programmed to replicate database changes received from a client over an asynchronous stream. In asynchronous mode, the server replicates transactions without acknowledgement. When a synchronization command is received, the server switches to synchronized mode, bringing the database to a consistent state and notifying the client. In synchronized mode, the server executes database commands and sends an acknowledgement to the client. Finally, replication resumes in asynchronous mode.
17. The method of claim 1 , further comprising: after detecting the synchronization condition, pausing, by the replication client, transmission of the plurality of change records; wherein resuming replication after acknowledgement is received by the replication client comprises the replication client resuming transmission of the plurality of change records.
In the method for database replication from a previous description, the replication client pauses transmission of change records after detecting the synchronization condition. Replication resumes after the client receives acknowledgement of the synchronized transactions. This prevents further asynchronous changes until synchronization is complete.
18. The method of claim 4 , wherein the change record for which asynchronous replication is not supported describes a schema change.
In the method for database replication from a previous description, the change record that triggers the switch to synchronized mode is one that describes a schema change. Schema changes often require synchronized replication to maintain data consistency.
19. The method of claim 1 , wherein detecting the synchronization condition comprises detecting, by the target database server, that an synchronized event has occurred that requires synchronization of the target database server and the replication client of the source database server before handling the synchronized event.
In the method for database replication from a previous description, detecting the synchronization condition includes the target database server detecting a synchronized event that requires synchronization before it can proceed.
20. The one or more computer-readable media of claim 8 : wherein detecting the synchronization condition comprises detecting one or more change records, corresponding to one the or more second transactions, for which asynchronous replication is not supported; wherein executing, by the target database server, the one or more database commands is performed after receiving the synchronization command over an asynchronous stream.
In the computer-readable medium from a previous description, the synchronization condition is detected by the replication client identifying one or more change records, corresponding to the one or more second transactions, for which asynchronous replication is not supported. The database commands are executed after receiving the synchronization command over the asynchronous stream.
21. The one or more computer-readable media of claim 20 , wherein the synchronization command is transmitted to the target database server over the asynchronous stream.
In the computer-readable medium from the previous description, the synchronization command is transmitted to the target database server over the asynchronous stream.
22. The one or more computer-readable media of claim 8 , wherein the one or more sequences of instructions include instructions that, when executed by the one or more processors, cause: after detecting the synchronization condition, pausing transmission of the plurality of change records; wherein resuming replication after acknowledgement is received comprises resuming transmission of the plurality of change records.
In the computer-readable medium from the previous description, the replication client pauses transmission of the plurality of change records after detecting the synchronization condition, and transmission is resumed after receiving acknowledgement of the one or more second transactions.
Cooperative Patent Classification codes for this invention. Click any code to explore related patents in that topic.
July 25, 2013
March 7, 2017
Browse 5M+ US patents with plain-English claim translations and AI-generated analysis.