Legal claims defining the scope of protection. Each claim is shown in both the original legal language and a plain English translation.
1. A system, process and method for detecting whether content within a given document is common to content within a plurality of other, existing documents, the system, process and method comprising: a character string recognizer for recognizing character strings in the content of the given document; a character string distinguisher for distinguishing main character strings and auxiliary character strings in the recognized character strings by reference to a closed list of main character strings; an encoder for encoding the content of the given document by assigning the main and auxiliary character strings differently using one or more digits; and a matcher for matching a plurality of n-digit streams from within the encoded content with any corresponding n-digit streams within previously-encoded content of the one or more other documents.
The invention is a system, process, and method to find common content between a given document and a set of existing documents. It works by first using a character string recognizer to identify strings within the given document. A character string distinguisher then categorizes those strings as either main or auxiliary strings, referencing a predefined list of main strings. The system then encodes the document's content by assigning different numerical digits to main and auxiliary strings. Finally, a matcher compares sequences of these digits (n-digit streams) from the encoded given document with corresponding streams in the pre-encoded existing documents to detect common content.
2. The system, process and method of claim 1 , wherein the documents comprise text written in one or more natural languages.
The content detection system, which recognizes and distinguishes character strings in a given document using an encoder to assign digits to main and auxiliary strings followed by a matcher to compare n-digit streams with other documents, operates on documents containing text written in one or more natural languages (e.g., English, Spanish).
3. The system, process and method of claim 2 , wherein the character string recognizer pre-filters the content of the given document to remove or replace any non-textual characters.
The content detection system, which recognizes and distinguishes character strings in text-based documents using an encoder and a matcher, includes a pre-filtering step within the character string recognizer. This pre-filter removes or replaces any non-textual characters (e.g., HTML tags, special symbols) from the given document's content before the character string recognition process. This ensures that only relevant text is processed.
4. The system, process and method of claim 2 , wherein at least some of the auxiliary character strings are pre-defined and comprise one or more of articles, conjunctions, prepositions and pronouns of the one or more natural languages.
The content detection system, which recognizes and distinguishes character strings in text-based documents using an encoder and a matcher, uses predefined auxiliary character strings, where at least some of these auxiliary strings are articles (e.g., "a", "the"), conjunctions (e.g., "and", "but"), prepositions (e.g., "in", "on"), and pronouns (e.g., "he", "she") from the natural language(s) of the documents being analyzed.
5. The system, process and method of claim 2 , wherein a plurality of sets of auxiliary character strings are pre-defined.
The content detection system, which recognizes and distinguishes character strings in text-based documents using an encoder and a matcher, uses a plurality of pre-defined sets of auxiliary character strings. This means that different sets of auxiliary character strings (e.g., stop words, common words) can be used, potentially tailored to different types of documents or different analysis goals.
6. The system, process and method of claim 5 , wherein the natural language of the given document is determined by reference to the respective set of pre-defined auxiliary character strings having greatest correlation.
The content detection system, which recognizes and distinguishes character strings in text-based documents using an encoder and a matcher with a plurality of pre-defined sets of auxiliary character strings, determines the natural language of the given document by identifying which set of pre-defined auxiliary character strings has the greatest correlation (i.e., the most matches) with the content of the document. This allows the system to automatically identify the language of the input document.
7. The system, process and method of claim 2 , wherein the closed list of main character strings comprises a list of stopwords and for at least one natural language, the stopwords are pre-defined.
The content detection system, which recognizes and distinguishes character strings in text-based documents using an encoder and a matcher, utilizes a closed list of main character strings that includes a list of stopwords (common words to be ignored). For at least one natural language, these stopwords are pre-defined, meaning they are determined in advance and used consistently for that language.
8. The system, process and method of claim 7 , wherein, for at least one natural language, the words within a reference corpus for the natural language are ranked by the vocabulary of collocations, and the list of stopwords comprises the highest ranked words constituting a pre-defined fraction (by volume) of the combined word frequency.
In the content detection system that uses pre-defined stopwords, the stopwords for at least one language are determined by ranking words within a reference corpus (a large body of text) for that language based on the vocabulary of collocations (words that frequently appear together). The stopwords list comprises the highest ranked words, constituting a predefined percentage of the overall word frequency in the corpus.
9. The system, process and method of claim 8 , wherein the pre-defined fraction comprises 50% (by volume) of the combined word frequency.
In the content detection system that ranks words by collocation frequency to determine stopwords, the pre-defined fraction of the combined word frequency that defines the stopword list is 50%. This means the most frequent words that account for 50% of the total word occurrences in the reference corpus are considered stopwords.
10. The system, process and method of claim 1 , wherein the matcher slides a window of n digits at an interval of m digits, where m is 1 or more, through the encoded content of the given document.
The content detection system's matcher, which compares encoded n-digit streams to find common content, operates by sliding a window of 'n' digits across the encoded content of the given document. This window moves at an interval of 'm' digits, where 'm' is 1 or more. If m=1, the window moves one digit at a time. This sliding window approach ensures that all possible n-digit sequences are compared.
11. The system, process and method of claim 1 , wherein the matcher post filters any matches to perform a similarity computation measure.
The content detection system's matcher, after finding potential matches of n-digit streams, includes a post-filtering step. This post-filtering applies a similarity computation measure to the matched content to refine the results and provide a more accurate assessment of similarity between the documents.
12. The system, process and method of claim 11 , wherein the similarity computation measure consists of a Dice coefficient or distance function, such as cosine distance function performed on the auxiliary character strings within the matched content.
The content detection system, including a matcher that performs post-filtering on matched content, uses a Dice coefficient or a distance function, like the cosine distance function, as its similarity computation measure. This calculation is performed specifically on the auxiliary character strings within the matched content, providing a measure of how similar the contexts of the matched segments are.
13. The system, process and method of claim 1 , wherein the digits comprise binary bits and the encoder encodes the content of the given document in binary code by assigning at least one bit string of one kind to the main character strings and at least one other bit string of other kind to the auxiliary character strings.
The content detection system, which encodes content by assigning digits to character strings, uses binary bits as the digits. The encoder encodes the content of the given document in binary code by assigning at least one bit string (sequence of 0s and 1s) to the main character strings and at least one *different* bit string to the auxiliary character strings. This binary encoding allows for efficient storage and comparison of the document content.
14. The system, process and method of claim 13 , wherein the encoder encodes the content of the given document in binary code by assigning a single binary bit of one kind (binary 0 or 1) to each main character string and a single binary bit of the other kind (binary 1 or 0) to each auxiliary character string.
The content detection system encodes character strings into binary code, assigning a single binary bit of one type (either 0 or 1) to each main character string, and a single binary bit of the opposite type (1 or 0) to each auxiliary character string. This simplified binary encoding uses a single bit to represent each main and auxiliary string, maximizing compression and speed.
15. The system, process and method of claim 13 , wherein the bit strings each comprise a plurality of binary bits, representing a plurality of symbols.
The content detection system's binary encoder, which assigns bit strings to character strings, uses bit strings that each comprise a *plurality* of binary bits, representing a *plurality* of symbols. Rather than using single bits, this approach employs multi-bit sequences, allowing each sequence to represent a different symbol or code.
16. The system, process and method of claim 15 , wherein the bit strings comprise 3 bits, representing octal or 4 bits, representing hex.
The content detection system's binary encoder uses bit strings comprised of either 3 bits (representing octal numbers or 8 symbols) or 4 bits (representing hexadecimal numbers or 16 symbols). This allows for more complex encoding compared to single-bit representation, providing more granular distinction between character strings.
17. The system, process and method of claim 13 , wherein the encoder encodes the content of the given document in binary code by assigning the bit string representing one symbol to each main character string and the bit strings representing the other symbols to auxiliary character strings.
The content detection system, which encodes content by assigning bit strings to character strings, encodes the content of the given document in binary code by assigning the bit string representing *one* symbol to each main character string and the bit strings representing the *other* symbols to auxiliary character strings. This means that all auxiliary character strings are differentiated from main strings but not necessarily from each other.
18. The system, process and method of claim 13 , wherein the encoder encodes the content of the given document in binary code by assigning the bit strings representing symbols to pre-defined fractions of the reference character strings and where the pre-defined fractions of the character strings are approximately or substantially equal by volume.
The content detection system encodes character strings into binary code using bit strings representing symbols. The system assigns bit strings representing these symbols to pre-defined *fractions* of the reference character strings, where these pre-defined fractions of character strings are approximately or substantially equal in volume (i.e., number of occurrences).
19. The system, process and method of claim 1 , further comprising: an indexer for indexing the encoded content of the one or more existing documents, wherein the indexer may comprise a sliding window of n digits which slides at a defined interval of m digits, where m is 1 or more, through each existing document to create an index which comprises an index address which is tagged with metadata identifying the reference document and the position of the index within the reference document.
The content detection system further includes an indexer for indexing the encoded content of the existing documents. The indexer uses a sliding window of 'n' digits, moving at an interval of 'm' digits (where 'm' is 1 or more) through each existing document. This creates an index where each entry comprises an index address tagged with metadata identifying the source document and the position of the index within the reference document.
20. The system, process and method of claim 1 , wherein the character string distinguisher uses the Van Rijsbergen or other pre-defined stopword list to identify the auxiliary character strings.
The character string distinguisher in the content detection system utilizes the Van Rijsbergen or another predefined stopword list to identify the auxiliary character strings. These stopword lists are standard resources defining common words (e.g., "the", "a", "is") that are typically filtered out during text analysis to improve performance and focus on more significant terms.
21. The system, process and method of claim 1 , wherein, in encoding the given document, the encoder assigns a different digit to each sub type of the main character strings and the main character strings are encoded using n digits, where n is in the range from 2 to 8 inclusive and n-1 digits are each used to represent 1/n by volume of a reference corpus and the 1 digit is associated with content which, when all auxiliary character strings are excluded, is not encountered in the reference corpus.
When encoding a given document, the system assigns a different digit to each sub-type of main character strings. The main character strings are encoded using 'n' digits, where 'n' is between 2 and 8 (inclusive). N-1 of these digits each represent 1/n by volume of a reference corpus. The remaining 1 digit is associated with content *not* encountered in the reference corpus when all auxiliary character strings are excluded.
22. The system, process and method of claim 1 , wherein the existing documents are in a plurality of different natural languages, each having a list of character strings comprising a pre-defined fraction by volume with highest collocate vocabulary in that language, and each list being representative of auxiliary character strings common to the language.
The content detection system works with existing documents in multiple natural languages. Each language has its own list of character strings comprising a pre-defined fraction by volume with the highest collocation vocabulary in that language. Each of these language-specific lists represents auxiliary character strings common to that language. This accounts for language-specific stop words and common terms.
23. The system, process and method of claim 22 , wherein, where an overlap occurs and an auxiliary character string is common to more than one of the languages, the language in which the auxiliary character string most frequently occurs is identified and the auxiliary character string in the list for that language retained and deleted from the lists in the other languages, thereby generating unique lists for the respective languages.
The content detection system handles overlaps between auxiliary character string lists in different languages. If an auxiliary character string is common to more than one language, the system identifies the language in which that string occurs most frequently. The system then retains the string in the list for *that* language and removes it from the lists of the *other* languages, generating unique lists for each language.
Unknown
September 12, 2017
Browse 5M+ US patents with plain-English claim translations and AI-generated analysis.