Predicting response to immunotherapy using computer extracted features of cancer nuclei from hematoxylin and eosin (HandE) stained images of non-small cell lung cancer (NSCLC)

This applicationis a reissue of U.S. Pat. No. 11,055,844, issued on Jul. 6, 2021, filed on Feb. 21, 2019, whichclaims the benefit of U.S. Provisional ApplicationNo.62/633,342 filedonFeb. 21, 2018, the disclosure ofwhich is incorporated by reference herein in its entirety.

This invention was made with government support underthe grant(s): 1U24CA199374-01, R01CA202752, R01CA202752-01A1, R01CA208236-01A1, R21CA179327-01, R21CA195152-01, R01 DK098503-02, 1 C06 RR12463-01 and NIH T32EB007509, awarded by the National Institutes of Health. Also PC120857, LC130463, and W81XWH-16-1-0329 awarded by the Department of Defense. The government has certain rights in the invention.W81XWH-16-1-0329, W81XWH-14-1-0323, W81XWH-13-1-0418, and CA179327, CA 195152, DK098503, CA199374, CA202752, CA208236, RR012463, and EB007509 awarded by the National Institutes of Health. The government has certain rights in the invention.

Immune checkpoint inhibitors are used in treating advanced stage non-small cell lung cancer (NSCLC). These drugs, including Nivolumab, target the programmed cell death protein 1 (PD-1) receptor or its ligand PD-L1. However, patients treated with immune checkpoint inhibitors have a response rate of only approximately 20%. The current gold standard biomarker, detection of tissue-based PD-L1 expression, is inadequate. It is thus crucial to identify which patients will derive maximal benefit from such treatments.

Embodiments predict response to immunotherapy in non-small cell lung cancer (NSCLC). Embodiments access a digitized hematoxylin and eosin (H&E) stained image of a region of tissue demonstrating NSCLC. The region of tissue includes a plurality of cellular nuclei. Embodiments may segment nuclear boundaries using a deep learning approach.

Embodiments extract a set of nuclear shape features and texture features from segmented cellular nuclei represented in the digitized H&E stained imagery of the region of tissue. The set of nuclear shape features and texture features may include a nuclear size feature, a nuclear area feature, a nuclear axis length feature, a nuclear perimeter feature, or a nuclear texture feature. Nuclear texture features may include, for example, a Haralick feature.

Embodiments further construct a nuclear cell graph (CG) based on the cellular nuclei represented in the digitized H&E stained image. In one embodiment, the cell graph is a global cell graph in which each nucleus represented in the digitized H&E stained image defines a node of the graph. Embodiments may define nodes on all the cellular nuclei represented in the digitized H&E image. Thus, embodiments may define nodes of the CG on different types of nuclei. For example, embodiments may define nodes on cancer cell nuclei and on tumor infiltrating lymphocytes, or on other types of cellular nuclei. Nodes may be connected based on distance metrics such as Euclidean Distance between nodes, or the L1 norm. In another embodiment, a threshold number of nuclei (e.g., 50%, 75%, or 90%) represented in the digitized H&E stained image may be employed to define nodes of the graph. The threshold number of nuclei may be user selectable, may be defined according to available computational resources, or may be defined according to a desired level of predictive accuracy.

Embodiments quantitatively evaluate the spatial arrangement of nuclei through the construction of a CG or CGs. A graph is a mathematical construct comprising of a finite sets of objects (nodes) that capture global and local relationships via pair-wise connections (edges) between the nodes. Graphs may be used to quantitatively characterize nuclear architecture in histopathological images by representing the nuclei as nodes and subsequently quantifying neighborhood relationships (e.g., proximity) and spatial arrangement between the nodes.

Embodiments represent centroids of each of, or a threshold number of, the cellular nuclei represented in the image as nodes of a graph. Nodes may be connected to others based on a weighted Euclidean norm where a weighting function favors connectivity between proximal nodes. In existing approaches, this may result in multiple disconnected subgraphs being generated. Embodiments construct a global CG without disconnected subgraphs. The threshold number of nuclei defined as nodes of a graph, may be selected based on a desired level of predictive accuracy, or on a desired use of computational resources, or on other criteria.

Embodiments compute a set of cell graph features based on the CG. The set of cell graph features capture tumor morphology within the microenvironment of the tumor. These features may include first-order statistics (e.g. mean, mode, median) of the representative descriptors. In one embodiment, the set of cell graph features may include a Delaunay side length disorder of the cells feature. The set of cell graph features may also include a Delaunay ratio of the minimum and maximum triangular areas formed by cells feature. The set of cell graph features may also include a number of possible triangles formed from cells (i.e., nodes) of the cell graph feature. Other cell graph features may be computed.

Embodiments provide the set of nuclear shape features and texture features, and the set of cell graph features, to a machine learning classifier trained to distinguish tissue that will respond to immunotherapy from tissue that will not respond to immunotherapy. In one embodiment, the machine learning classifier is a quadratic discriminant analysis (QDA) classifier. Embodiments receive a probability of response computed by the machine learning classifier. The machine learning classifier computes the probability based on the set of nuclear shape features and texture features, and the set of cell graph features. Embodiments classify the region of tissue as likely to experience response to immunotherapy, or unlikely to experience response to immunotherapy, based, at least in part, on the probability of response. Immunotherapy may include Nivolumab immunotherapy, pembrolizumab, ateziolizumab, or other type of checkpoint inhibitor immunotherapy. Response may include pathological complete response (pCR), or other type of response.

In one embodiment, digitized images of pre-treatment H&E stained tissue slides of pre-treatment tumor biopsies are acquired of a cohort of fifty six (56) patients, from two different institutions. The patients demonstrate NSCLC and were treated with Nivolumab or other form of NSCLC immunotherapy. The cohort of 56 patients was split into two categories: responders, and non-responders. Membership in a category (e.g., responder, non-responder) was determined by clinical involvement and radiological assessment using the RECIST criteria. In another embodiment, membership in a category (e.g., responder, non-responder) may be determined using another, different criteria. In one embodiment, the cohort is randomly divided into a training set (n=32) and a testing set (n=24). In one embodiment, 245radiomicfeatures were extracted from tumor nuclei, and included features that characterize shape and texture of the tumor nuclei represented in the digitized images. Nuclei may be annotated automatically, using for example, a deep learning approach or a watershed approach, or manually by expert human pathologists on the digitized H&E images.

A statistical feature selection method was employed to determine the top five most discriminative features from the training set. In one embodiment, a minimum redundancy, maximum relevance (mRMR) feature selection technique is employed. In another embodiment, other feature selection techniques may be employed. The top five features in this embodiment capture the spatial arrangement of nuclei and variance in nuclear shape and chromatin structure.illustrates box plotsand. Box plotillustrates the top five most significant (i.e., discriminative) features associated with responders. The vertical axes in box plotsandrepresent the continuous feature values shown as a distribution over the sample population. Box plotincludes plots for a side length disorder of a Delaunay triangulation feature, a ratio of minimum and maximum triangular areas formed by nodes of the CG, and a number of possible polygons formed by nodes of the CG. Box plotalso includes plots for a standard deviation of the fractal dimension of a nucleus feature, and a mean of a tensor contrast entropy of cellular nuclei feature. Box plotillustrates the top five most significant (i.e., discriminative) features associated with non-responders Box plotincludes plots for a side length disorder of a Delaunay triangulation feature, a ratio of minimum and maximum triangular areas formed by nodes of the CG, and a number of possible polygons formed by nodes of the CG. Box plotalso includes plots for a standard deviation of the fractal dimension of a nucleus feature, and a mean of a tensor contrast entropy of cellular nuclei feature.

Some portions of the detailed descriptions that follow are presented in terms of algorithms and symbolic representations of operations on data bits within a memory. These algorithmic descriptions and representations are used by those skilled in the art to convey the substance of their work to others. An algorithm, here and generally, is conceived to be a sequence of operations that produce a result. The operations may include physical manipulations of physical quantities. Usually, though not necessarily, the physical quantities take the form of electrical or magnetic signals capable of being stored, transferred, combined, compared, and otherwise manipulated in a logic or circuit, and so on. The physical manipulations create a concrete, tangible, useful, real-world result.

It has proven convenient at times, principally for reasons of common usage, to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, and so on. It should be borne in mind, however, that these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless specifically stated otherwise, it is appreciated that throughout the description, terms including processing, computing, calculating, determining, and so on, refer to actions and processes of a computer system, logic, circuit, processor, or similar electronic device that manipulates and transforms data represented as physical (electronic) quantities.

Example methods and operations may be better appreciated with reference to flow diagrams. While for purposes of simplicity of explanation, the illustrated methodologies are shown and described as a series of blocks, it is to be appreciated that the methodologies are not limited by the order of the blocks, as some blocks can occur in different orders and/or concurrently with other blocks from that shown and described. Moreover, less than all the illustrated blocks may be required to implement an example methodology. Blocks may be combined or separated into multiple components. Furthermore, additional and/or alternative methodologies can employ additional, not illustrated blocks.

is a flow diagram of example operationsthat may be performed by a processor to predict response to immunotherapy in NSCLC. A processor(s) may include any combination of general-purpose processors and dedicated processors (e.g., graphics processors, application processors, etc.). The processors may be coupled with or may include memory or storage and may be configured to execute instructions stored in the memory or storage to enable various apparatus, applications, or operating systems to perform the operations. The memory or storage devices may include main memory, disk storage, or any suitable combination thereof. The memory or storage devices may include, but are not limited to any type of volatile or non-volatile memory such as dynamic random access memory (DRAM), static random-access memory (SRAM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), Flash memory, or solid-state storage.

The set of operationsincludes, at, accessing a digitized image of a region of tissue (ROT) demonstrating cancerous pathology. The ROT includes a plurality of cellular nuclei. The digitized image includes a plurality of pixels, a pixel having an intensity. In one embodiment, the digitized image is a digitized image of a H&E stained slide of region of tissue demonstrating non-small cell lung cancer (NSCLC) scanned at 20× magnification. While digitized H&E images scanned at 20× magnification are described in this example, images having other imaging parameters or acquired using other imaging modalities may be employed. Accessing the digitized image includes acquiring electronic data, reading from a computer file, receiving a computer file, reading from a computer memory, or other computerized activity not practically performed in a human mind.

Operationsalso includes, at, segmenting a plurality of cellular nuclei represented in the digitized image. In one embodiment, segmenting the plurality of cellular nuclei represented in the digitized image includes segmenting the plurality of cellular nuclei using a deep learning approach. For example, embodiments may employ a convolutional neural network (CNN) configured to segment nuclei from non-nuclei portions of the digitized image. Segmenting the plurality of cellular nuclei includes acquiring electronic data, reading from a computer file, receiving a computer file, reading from a computer memory, or other computerized activity not practically performed in a human mind.illustrates a region of tissuedemonstrating NSCLC.further illustrates a sub-sectionof the region of tissue. A magnified viewof subsectionfurther illustrates segmented cellular nuclei. In embodiments described herein, segmented cellular nucleimay be, for example a cancerous nucleus, a tumor infiltrating lymphocyte (TIL), or other type of cellular nucleus.

In one embodiment, a watershed-based technique is used for automatically detecting and segmenting members of the plurality of cellular nuclei represented in the digitized image. This technique applies a set of mathematical operations, including fast radial symmetry transform and regional minima, at different scales (e.g., 5×, 10× and 20×) to identify candidate locations for nuclei. This technique improves on those employed by existing approaches to segmenting nuclei by being computationally simpler and faster. This technique also facilitates the adjustment and fine-tuning of parameters with greater simplicity than techniques used by existing approaches, thereby providing the technical effect of improving the performance of computers, systems, or other apparatus on which embodiments are implemented. In one embodiment, cellular nuclei represented in the digitized H&E image are already segmented, and thus in one embodiment, the operations atmay not need to be performed.

Operationsalso includes, at, extracting a set of nuclearradiomicfeatures from the plurality of segmented cellular nuclei. In one embodiment, the set of nuclearradiomicfeatures includes at least one of a nuclear size feature, a nuclear area feature, a nuclear axis length feature, a nuclear perimeter feature, or a nuclear texture feature. In one embodiment, the set of nuclearradiomicfeatures includes a standard deviation of the fractal dimension of a nucleus feature, and a mean of a tensor contrast entropy of cellular nuclei feature. Extracting the set of nuclearradiomicfeatures includes acquiring electronic data, reading from a computer file, receiving a computer file, reading from a computer memory, or other computerized activity not practically performed in a human mind.illustrates a region of tissuedemonstrating NSCLC. Region of tissueincludes a plurality of cellular. A magnified view of a member of the plurality of cellular nucleiis illustrated. Nuclearradiomicfeatures (e.g., shape features, texture features) associated with the member of the plurality of cellular nucleiare illustrated at.

Operationsalso includes, at, generating at least one nuclear cell graph (CG) based on the plurality of segmented cellular nuclei. In one embodiment, a node of the at least one nuclear CG is defined on a centroid of a member of the plurality of cellular nuclei. A first node is connected to a second, different node based on a Euclidean distance between the first node and the second node. In another embodiment, the centroid of a local nuclei cluster is used as a node, and a plurality of nodes is used to construct the global CG. The probability a first node will be linked with a second, different node is based on an exponentially decaying function of the Euclidean distance between the nodes. In one embodiment, the at least one nuclear CG is a global CG. In another embodiment, the digitized image may include more than one tumor region. In that case, where there is more than one tumor region represented in the digitized image, embodiments may construct more than one CG. For example, embodiments may construct one CG for each tumor region, respectively.illustrates a region of tissue. Region of tissueincludes more than one tumor region. For example, region of tissue includes a first tumor region, a second, different tumor region, and third, different tumor region. Generating the at least one nuclear CG includes acquiring electronic data, reading from a computer file, receiving a computer file, reading from a computer memory, or other computerized activity not practically performed in a human mind.

Operationsalso includes, at, computing a set of CG features based on the at least one nuclear CG. In one embodiment, the set of CG features includes at least one of a Delaunay triangulation feature or a Voronoi feature. In one embodiment, the set of CG features includes a side length disorder of a Delaunay triangulation feature, a ratio of minimum and maximum triangular areas formed by nodes of the CG, and a number of possible polygons formed by nodes of the CG. In this embodiment, a polygon is a triangle. Computing the set of CG features includes acquiring electronic data, reading from a computer file, receiving a computer file, reading from a computer memory, or other computerized activity not practically performed in a human mind.

illustrates cellular nuclei graphs in NSCLC tissue. A region of tissuethat responded to immunotherapy is illustrated. Region of tissueincludes a plurality of cellular nuclei. A global nuclear CGassociated with the region of tissueis illustrated. A region of tissuethat did not respond to immunotherapy is illustrated. Region of tissueincludes a plurality of cellular nuclei. A global nuclear CGassociated with the region of tissueis also illustrated.

Operationsalso includes, at, providing the set of nuclearradiomicfeatures and the set of CG features to a machine learning classifier. In one embodiment, the machine learning classifier is a QDA classifier. In another embodiment, other types of machine learning classifiers, including a linear discriminant analysis (LDA) classifier, a random forest classifier, or a deep learning classifier, including a CNN, may be employed. Providing the set of nuclearradiomicfeatures and the set of CG features to the machine learning classifier includes acquiring electronic data, reading from a computer file, receiving a computer file, reading from a computer memory, or other computerized activity not practically performed in a human mind.

Operationsalso includes, at, receiving, from the machine learning classifier, a probability that the ROT will respond to immunotherapy. The machine learning classifier computes the probability based, at least in part, on the set of nuclearradiomicfeatures and the set of CG features. Receiving the probability includes acquiring electronic data, reading from a computer file, receiving a computer file, reading from a computer memory, or other computerized activity not practically performed in a human mind.

Operationsalso includes, at, generating a classification of the ROT as a responder or non-responder based on the probability. The classification is generated, based, at least in part, on the probability. For example, embodiments may classify the region of tissue as likely to respond to immunotherapy when the probability >=0.5, and may classify the region of tissue as unlikely to respond to immunotherapy when the probability <0.5. Other classification schemes may be employed. In one embodiment, the classification is further based on the digitized image. Generating the classification includes acquiring electronic data, reading from a computer file, receiving a computer file, reading from a computer memory, or other computerized activity not practically performed in a human mind.

Operationsfurther includes, atdisplaying the classification. Displaying the classification may include displaying the classification on a computer monitor, a smartphone display, a tablet display, or other displays. Displaying the classification may also include printing the classification. Displaying the classification may also include controlling an immunotherapy response prediction system, a personalized medicine system, a monitor, or other display, to display operating parameters or characteristics of a machine learning classifier, during both training and testing of the machine learning classifier, or during clinical operation of the machine learning classifier. By displaying the classification example embodiments provide a timely and intuitive way for a human medical practitioner to more accurately classify a region of tissue represented in digitized imagery as likely to respond to immunotherapy, or unlikely to respond to immunotherapy, thus improving on existing approaches to predicting response to immunotherapy. Embodiments may further display operating parameters of the machine learning classifier. Embodiments may further display the set of CG features, the set of nuclearradiomicfeatures, the at least one CG, or the digitized image.

Whileillustrates various actions occurring in serial, it is to be appreciated that various actions illustrated incould occur substantially in parallel. By way of illustration, a first process could involve accessing a digitized H&E stained image, a second process could involve extractingradiomicfeatures from a cellular nucleus represented in the image, and a third process could involve constructing a cell graph. While three processes are described, it is to be appreciated that a greater or lesser number of processes could be employed and that lightweight processes, regular processes, threads, and other approaches could be employed.

illustrates a set of operationsthat is similar to operationsbut that includes additional details and elements. Operationsinclude, at, training the machine learning classifier to compute the probability that a region of tissue will respond to immunotherapy.illustrates operationsfor training the machine learning classifier. In one embodiment, operationsinclude, at, accessing a set of digitized images of H&E stained slides of NSCLC tissue scanned at 20× magnification, where a digitized image includes a plurality of pixels, a pixel having an intensity, where the set of digitized images includes images of patients who had immunotherapy, where a response status of the patient is known. Members of the set of digitized images may be acquired from different institutions, may be acquired using different scanners, different staining parameters, or at different magnifications.

Operationsalso includes, at, extracting a set of nuclearradiomicfeatures from the set of digitized images. Extracting the set of nuclearradiomicfeatures includes acquiring electronic data, reading from a computer file, receiving a computer file, reading from a computer memory, or other computerized activity not practically performed in a human mind. The set of nuclearradiomicfeatures may include at least one of a nuclear size feature, a nuclear area feature, a nuclear axis length feature, a nuclear perimeter feature, or a nuclear texture feature. In one embodiment, the set of nuclearradiomicfeatures includes a standard deviation of the fractal dimension of a nucleus feature, and a mean of a tensor contrast entropy of cellular nuclei feature.

Operationsalso includes, at, extracting a set of cellular graph (CG) features from the set of digitized images. In one embodiment, the set of CG features includes at least one of a Delaunay triangulation feature or a Voronoi feature. In one embodiment, the set of CG features includes a side length disorder of a Delaunay triangulation feature, a ratio of minimum and maximum triangular areas formed by nodes of the CG, and a number of possible polygons formed by nodes of the CG. In this embodiment, a polygon is a triangle. Extracting the set of CG features may include constructing at least one CG based on segmented cellular nuclei represented in a member of the set of digitized images according to embodiments described herein. Extracting the set of CG features includes acquiring electronic data, reading from a computer file, receiving a computer file, reading from a computer memory, or other computerized activity not practically performed in a human mind.

Operationsalso includes, at, generating a set of discriminative features by selecting a threshold number of the most discriminatoryradiomicfeatures and cellular graph features that discriminate response to immunotherapy from non-response to immunotherapy. In one embodiment, the set of discriminative features is selected using an mRMR feature selection approach. In another embodiment, another, different feature selection approach may be employed. In one embodiment, the set of discriminative features includes five features. Generating the set of discriminative features includes acquiring electronic data, reading from a computer file, receiving a computer file, reading from a computer memory, or other computerized activity not practically performed in a human mind.

Operationsalso includes, at, generating a training set. The training set is a first subset of the set of digitized images. The training set includes at least one image acquired of a patient that responded to immunotherapy, and at least one image acquired of a patient that did not respond to immunotherapy. Generating the training set includes acquiring electronic data, reading from a computer file, receiving a computer file, reading from a computer memory, or other computerized activity not practically performed in a human mind.

Operationsalso includes, at, generating a testing set. The testing set is a second, disjoint subset of the set of digitized images. The testing set includes at least one image acquired of a patient that responded to immunotherapy, and at least one image acquired of a patient that did not respond to immunotherapy. Generating the testing set includes acquiring electronic data, reading from a computer file, receiving a computer file, reading from a computer memory, or other computerized activity not practically performed in a human mind.

Operationsalso includes, at, training the machine learning classifier to generate a probability of response using the training set and the set of discriminative features. Training the machine learning classifier may also include determining which features are most discriminative in distinguishing tissue likely to respond to from tissue unlikely to respond. Embodiments may adjust the set of discriminative features based on a desired training time, a desired predictive accuracy, or desired execution time. Adjusting the set of discriminative features may include selecting more than five features, or fewer than five features, or selecting different features for inclusion in the set of discriminative features. Training the machine learning classifier may also include determining settings outside the machine learning classifier architecture but relevant to its learning behavior. Training the machine learning classifier includes acquiring electronic data, reading from a computer file, receiving a computer file, reading from a computer memory, or other computerized activity not practically performed in a human mind.

Operationsfurther includes, at, testing the machine learning classifier using the testing set and the set of discriminative features. The machine learning classifier is, in one embodiment, evaluated using a concordance index. Testing the machine learning classifier includes acquiring electronic data, reading from a computer file, receiving a computer file, reading from a computer memory, or other computerized activity not practically performed in a human mind.

Returning to FIG.10, the set of operationsmay also include, at, generating a personalized cancer treatment plan. The personalized cancer treatment plan may be generated based, at least in part, on the classification and at least one of the probability, the set of nuclearradiomicfeatures, the set of CG features, or the digitized image. The personalized cancer treatment plan may be generated for the patient of whom the digitized image was acquired based, at least in part, on the classification, the digitized image, or the set ofradiomicfeatures. Defining a personalized cancer treatment plan facilitates delivering a particular treatment that will be therapeutically active to the patient, while minimizing negative or adverse effects experienced by the patient. For example, the personalized cancer treatment plan may suggest a surgical treatment, may define an immunotherapy agent dosage or schedule, or a chemotherapy agent dosage or schedule, for a patient identified as likely to respond to immunotherapy. For a patient classified as unlikely to respond to immunotherapy, other treatments may be suggested. Generating the personalized cancer treatment plan includes acquiring electronic data, reading from a computer file, receiving a computer file, reading from a computer memory, or other computerized activity not practically performed in the human mind. The set of operationsmay further include, at, displaying the personalized cancer treatment plan.

In one example, a method may be implemented as computer executable instructions. Thus, in one example, a computer-readable storage device may store computer executable instructions that if executed by a machine (e.g., computer, processor) cause the machine to perform methods or operations described or claimed herein including operations,, or, method, or any other methods or operations described herein. While executable instructions associated with the listed methods are described as being stored on a computer-readable storage device, it is to be appreciated that executable instructions associated with other example methods or operations described or claimed herein may also be stored on a computer-readable storage device. In different embodiments the example methods or operations described herein may be triggered in different ways. In one embodiment, a method or operation may be triggered manually by a user. In another example, a method or operation may be triggered automatically.

Improved prediction of response may produce the technical effect of improving the administration of chemotherapy or immunotherapy, by increasing the accuracy of and decreasing the time required to determine if a patient is likely or unlikely to respond. Treatments and resources, including expensive immunotherapy or chemotherapy agents may be more accurately tailored to patients with a likelihood of benefiting from said treatments and resources, including responding to immunotherapy, so that more appropriate treatment protocols may be employed, and expensive resources are not wasted, when digitized H&E images are more accurately and more quickly assessed for likelihood of response. Controlling a response prediction apparatus based on improved, more accurate analysis of digitized H&E images further improves the operation of the system, processor, or apparatus, since the accuracy of the system, processor, or apparatus is increased and unnecessary operations will not be performed. Embodiments described herein, including at least operations,, or, method, or apparatusor, resolve features extracted from digitized H&E imagery at a higher order or higher level than a human can resolve in the human mind or with pencil and paper. For example, properties of the digitized H&E image that are not perceivable by the human eye may be detected by embodiments.Radiomic featuresFeaturesgenerated by embodiments are not properties of tumoral tissue that are perceivable by the human eye, and their computation is not practically performed in the human mind. Nuclear cell graphs are also not a property of tissue represented in the digitized H&E imagery. A machine learning classifier as described herein may not be implemented in the human mind or with pencil and paper. Embodiments thus perform actions, steps, processes, or other actions that are not practically performed in the human mind, at least because they require a processor or circuitry to access digitized images stored in a computer memory and to extract or compute features that are based on the digitized images and not on properties of tissue or the images that are perceivable by the human eye. Embodiments described herein use a combined order of specific rules, elements, operations, or components that render information into a specific format that is then used and applied to create desired results more accurately, more consistently, and with greater reliability than existing approaches, thereby producing the technical effect of improving the performance of the machine, computer, or system with which embodiments are implemented.

illustrates area under the receiver operating characteristic (AUC) curves for embodiments described herein.illustrates the AUC curvefor embodiments trained on a training set as described herein.also illustrates the AUC curvefor embodiments tested on a testing set as described herein. Embodiments predict response to immunotherapy based onradiomicand global graph features extracted from digitized H&E stained imagery with an AUC of at least 0.65, thus improving on existing approaches which may predict response to immunotherapy with less accuracy.

illustrates an example apparatus. Apparatusmay be configured to predict response to immunotherapy in patients demonstrating NSCLC, including early-stage NSCLC. Apparatusincludes a processor. Apparatusalso includes a memory. Processormay, in one embodiment, include circuitry such as, but not limited to, one or more single-core or multi-core processors. Processormay include any combination of general-purpose processors and dedicated processors (e.g., graphics processors, application processors, etc.). The processors may be coupled with or may include memory (e.g. memory) or storage and may be configured to execute instructions stored in the memoryor storage to enable various apparatus, applications, or operating systems to perform the operations. Memoryis configured to store a digitized image of a region of tissue demonstrating NSCLC. The digitized image has a plurality of pixels, a pixel having an intensity. Memorymay be further configured to store a training set of digitized images, or a testing set of digitized images. Memorymay be further configured to store metadata associated with digitized images, including response status data, overall survival (OS) data, or disease free survival (DFS) data associated with patients of whom the imagery is acquired.

Apparatusalso includes an input/output (I/O) interface, a set of circuits, and an interfacethat connects the processor, the memory, the I/O interface, and the set of circuits. I/O interfacemay be configured to transfer data between memory, processor, circuits, and external devices, for example, a digital whose slide scanner, an immunotherapy response prediction system, or a personalized medicine system.

The set of circuitsincludes an image acquisition circuit, aradiomicfeature circuit, a nuclear cell graph (CG) circuit, an immunotherapy response prediction circuit, and a display circuit. Image acquisition circuitis configured to access a digitized image of a region of tissue (ROT) demonstrating cancerous pathology, where the ROT includes a plurality of cellular nuclei. The digitized image has a plurality of pixels, a pixel having an intensity. In one embodiment the digitized image is a digitized image of an H&E stained slide of a region of tissue demonstrating NSCLC scanned at 20× magnification. In another embodiment, other, different magnification levels, or other, different types of image of tissue demonstrating NSCLC may be accessed or employed. Accessing the digitized image may include accessing a digitized image stored in memory. In one embodiment, accessing the digitized image may include accessing a digitized image stored in a data storage device, including a hard disk drive, a solid state device, a tape drive, or accessing a radiological image over a local area network. Accessing the digitized image includes acquiring electronic data, reading from a computer file, receiving a computer file, reading from a computer memory, or other computerized activity not practically performed in the human mind.

Image acquisition circuitis further configured to segment a plurality of cellular nuclei represented in the digitized image. In one embodiment, image acquisition circuitis configured to segment the plurality of cellular nuclei using a deep learning approach. In another embodiment, image acquisition circuitis configured to segment the plurality of cellular nuclei using a different segmentation technique. In another embodiment, image acquisition circuitis configured to access a digitized image in which a member of the plurality of cellular nuclei is already segmented.

Radiomic featureFeaturescircuitis configured to extract a set of nuclearradiomicfeatures from the plurality of segmented cellular nuclei. In one embodiment, the set of nuclearradiomicfeatures includes a standard deviation of the fractal dimension of a nucleus feature, and a mean of a tensor contrast entropy of cellular nuclei feature. In another embodiment, the set of nuclearradiomicfeatures includes at least one of a nuclear size feature, a nuclear area feature, a nuclear axis length feature, a nuclear perimeter feature, or a nuclear texture feature. In another embodiment, the set of nuclearradiomicfeatures may include other, differentradiomicfeatures.

Nuclear CG circuitis configured to generate at least one nuclear CG based on the plurality of segmented cellular nuclei. A node of the at least one nuclear CG is defined on a centroid of a member of the plurality of cellular nuclei. A first node is connected to a second, different node based on a Euclidean distance between the first node and the second node. In one embodiment, nuclear CG circuitis configured to group individual cell nuclei into clusters. The centroid of a cluster is used as a node and used to form a globally connected graph, where the probability of a link between nodes is inversely proportional to the Euclidean distance between nodes.

Nuclear CG circuitis further configured to compute a set of CG features based on the at least one nuclear CG. In one embodiment, the set of CG features includes a side length disorder of a Delaunay triangulation, a ratio of minimum and maximum triangular areas formed by nodes of the CG, and a number of possible triangles formed by nodes of the CG. In another embodiment, the set of CG features includes at least one of a Delaunay triangulation feature or a Voronoi feature. In another embodiment, the set of CG features may include other, different CG features, or statistical features computed based on the at least one nuclear CG.

Immunotherapy response classification circuitis configured to compute a probability that the ROT will respond to immunotherapy based, at least in part, on the set of nuclearradiomicfeatures and the set of CG features. Immunotherapy response classification circuitis further configured to generate a classification of the ROT as a responder or non-responder based on the probability. In one embodiment, immunotherapy response classification circuitis configured to compute the probability that the region of tissue will respond to immunotherapy using a quadratic discriminant analysis (QDA) machine learning approach. In another embodiment, immunotherapy response classification circuitis configured to compute the probability using another, different machine learning approach (e.g., LDA, random forest, neural networks). Immunotherapy response classification circuitmay further be configured to compute a probability that the patient of whom the digitized image is acquired will respond to immunotherapy, or to generate a patient-wise classification of the patient as a responder or non-responder, based, at least in part, on the probability.

Display circuitis configured to display the classification. Display circuitis further configured to display at least one of the probability, the set of nuclearradiomicfeatures, the set of CG features, the CG, or the digitized image. Displaying the classification or at least one of the probability, the set of nuclearradiomicfeatures, the set of CG features, the CG, or the digitized image may also include printing the classification or at least one of the probability, the set of nuclearradiomicfeatures, the set of CG features, the CG, or the digitized image.

illustrates an example apparatusthat is similar to apparatusbut that includes additional details and elements. In one embodiment, apparatusincludes a training circuit. Training circuitmay be configured to train immunotherapy response classification circuit, a machine learning classifier, including a QDA, to classify a region of tissue demonstrating NSCLC according to techniques described herein. In one embodiment, training circuitis configured to access a set of digitized H&E stained images of tissue demonstrating NSCLC pathology, and where the tissue includes a tumoral region. A digitized H&E stained image includes a plurality of pixels, a pixel having an intensity, where a response status for each patient, respectively, is known.