evaluate

STANDS provides an evaluate function for various metrics.

Modules:

Name	Description
evaluate	Calculate various metrics (including SGD).

evaluate

evaluate(
    metrics: Sequence[metrics_list],
    y_true=None,
    y_score=None,
    y_pred=None,
    adata: Optional[AnnData] = None,
    batchid: Optional[str] = None,
    typeid: Optional[str] = None,
    emb: Optional[str] = None,
    clustid: Optional[str] = None,
    spaid: Optional[str] = None,
    **kwargs
)

Evaluate performance metrics based on specified evaluation metrics. Different metrics require different parameters. Here is a description of the metrics that can be calculated and the parameters they require.

Functions:

Name	Description
AUC	y_true, y_pred/y_score
Precision	y_true, y_pred/y_score
Recall	y_true, y_pred/y_score
F1	y_true, y_pred/y_score
F1*NMI	y_true, y_pred/y_score, adata, typeid, clustid
SGD_degree	adata, spaid, y_true, y_pred/y_score
SGD_cc	adata, spaid, y_true, y_pred/y_score
ARI	adata, typeid, clustid
NMI	adata, typeid, clustid
ASW_type	adata, typeid, batchid, (Optional: emb)
1-ASW_batch	adata, typeid, batchid, (Optional: emb)
BatchKL	adata, batchid, (Optional: emb)
iLISI	adata, batchid, (Optional: emb)
cLISI	adata, typeid, (Optional: emb)

Parameters:

Name	Type	Description	Default
metrics	Sequence[str]	List of evaluation metrics to compute.	required
y_true	Optional[Union[Series, ndarray]]	True labels.	None
y_score	Optional[Union[Series, ndarray]]	Predicted scores or probabilities.	None
y_pred	Optional[Union[Series, ndarray]]	Predicted labels.	None
adata	Optional[AnnData]	Annotated data containing embeddings or clusters.	None
batchid	Optional[str]	Batch ID key in adata.obs for batch information.	None
typeid	Optional[str]	Type ID key in adata.obs for type information.	None
emb	Optional[str]	Key for embeddings in adata.obsm.	None
clustid	Optional[str]	Cluster ID key in adata.obs for clustering information.	None
spaid	Optional[str]	Spatial coordinates ID key in adata.obsm (for SGD_degree & SGD_cc metrics).	None

Other Parameters:

Name	Type	Description
n_neighbors	int	Number of neighbors for SGD KNN graph.
bins	int	Number of equal-width bins in the given range when calculating SGD_cc.
num_bootstrap_samples	int	Number of bootstrap samples for distribution estimation.
sigma	int	Sigma parameter for Gaussian Earth Mover's Distance.

Returns:

Type	Description
Union[Tuple[float], float]	Depending on the number of specified metrics, returns a tuple of metric values or a single metric value.

Raises:

Type	Description
RuntimeError	In the anomaly detection, it doesn't specify y_score or y_pred.

Note

SGD_degree & SGD_cc are available for both anomaly detection and subtyping tasks. They will automatically determine the category based on the types of anomalies in y_true eliminating the need for additional parameters to specify whether it is the subtyping task.

Source code in src\stands\evaluate\eval.py

@clear_warnings
def evaluate(metrics: Sequence[metrics_list],
             y_true=None, y_score=None, y_pred=None,
             adata: Optional[ad.AnnData]=None,
             batchid: Optional[str]=None, typeid: Optional[str]=None,
             emb: Optional[str] = None, clustid: Optional[str] = None,
             spaid: Optional[str] = None, **kwargs):
    """
    Evaluate performance metrics based on specified evaluation metrics.
    Different metrics require different parameters.
    Here is a description of the metrics that can be calculated and the parameters they require.

    Functions:
        AUC: `y_true`, `y_pred`/`y_score`
        Precision: `y_true`, `y_pred`/`y_score`
        Recall: `y_true`, `y_pred`/`y_score`
        F1: `y_true`, `y_pred`/`y_score`
        F1*NMI: `y_true`, `y_pred`/`y_score`, `adata`, `typeid`, `clustid`
        SGD_degree: `adata`, `spaid`, `y_true`, `y_pred`/`y_score`
        SGD_cc: `adata`, `spaid`, `y_true`, `y_pred`/`y_score`
        ARI: `adata`, `typeid`, `clustid`
        NMI: `adata`, `typeid`, `clustid`
        ASW_type: `adata`, `typeid`, `batchid`, (Optional: `emb`)
        1-ASW_batch: `adata`, `typeid`, `batchid`, (Optional: `emb`)
        BatchKL: `adata`, `batchid`, (Optional: `emb`)
        iLISI: `adata`, `batchid`, (Optional: `emb`)
        cLISI: `adata`, `typeid`, (Optional: `emb`)

    Parameters:
        metrics (Sequence[str]): List of evaluation metrics to compute.
        y_true (Optional[Union[pd.Series, np.ndarray]]): True labels.
        y_score (Optional[Union[pd.Series, np.ndarray]]): Predicted scores or probabilities.
        y_pred (Optional[Union[pd.Series, np.ndarray]]): Predicted labels.
        adata (Optional[ad.AnnData]): Annotated data containing embeddings or clusters.
        batchid (Optional[str]): Batch ID key in adata.obs for batch information.
        typeid (Optional[str]): Type ID key in adata.obs for type information.
        emb (Optional[str]): Key for embeddings in adata.obsm.
        clustid (Optional[str]): Cluster ID key in adata.obs for clustering information.
        spaid (Optional[str]): Spatial coordinates ID key in adata.obsm (for SGD_degree & SGD_cc metrics).

    Other Parameters:
        n_neighbors (int): Number of neighbors for SGD KNN graph.
        bins (int): Number of equal-width bins in the given range when calculating SGD_cc.
        num_bootstrap_samples (int): Number of bootstrap samples for distribution estimation.
        sigma (int): Sigma parameter for Gaussian Earth Mover's Distance.

    Returns:
        (Union[Tuple[float], float]): Depending on the number of specified metrics, returns a tuple of metric values or a single metric value.

    Raises:
        RuntimeError: In the anomaly detection, it doesn't specify `y_score` or `y_pred`.

    Note:
        SGD_degree & SGD_cc are available for both anomaly detection and subtyping tasks. 
        They will automatically determine the category based on the types of anomalies in y_true
        eliminating the need for additional parameters to specify whether it is the subtyping task.
    """
    data = {}

    if  y_true is not None:
        y_true = pd.Series(y_true)

        if y_score is not None:
            y_score = pd.Series(y_score)
            ratio = 100.0 * len(np.where(y_true == 0)[0]) / len(y_true)
            thres = np.percentile(y_score, ratio)
            y_pred = (y_score >= thres).astype(int)
            y_true = y_true.astype(int)
        elif y_pred is not None:
            y_pred = pd.Series(y_pred).astype(int)
        else:
            raise RuntimeError('Please input y_score or y_pred!')

        data.update({'y_true': y_true, 'y_score': y_score, 'y_pred': y_pred})

        # for SGD_degree and SGD_cc metrics
        if (adata is not None) and (spaid is not None):
            data.update({'spatial': adata.obsm[spaid]})

    if adata is not None:
        if emb is not None:
            correct = adata.obsm[emb]
            data.update({'correct': correct})
        elif ['ASW_type', '1-ASW_batch', 'BatchKL', 'iLISI', 'cLISI'] in metrics:
            sc.tl.tsne(adata, random_state=0, use_fast_tsne=False)
            correct = adata.obsm['X_tsne']
            data.update({'correct': correct})

        if batchid is not None:
            _, idx = np.unique(adata.obs[batchid].values, return_inverse=True)
            data.update({'batch': idx})
        if typeid is not None:
            _, idx = np.unique(adata.obs[typeid].values, return_inverse=True)
            data.update({'type': idx})
        if clustid is not None:
            _, idx = np.unique(adata.obs[clustid].values, return_inverse=True)
            data.update({'cluster': idx})

    method = {
        'AUC': eval_AUC,
        'Precision': eval_P, 
        'Recall': eval_R, 
        'F1': eval_F1,
        'F1*NMI': evla_F1NMI,
        'ARI': eval_ARI,
        'NMI': eval_NMI,
        'ASW_type': eval_ASW_type,
        '1-ASW_batch': eval_ASW_batch,
        'BatchKL': eval_BatchKL,
        'iLISI': eval_iLISI,
        'cLISI': eval_cLISI,
        'SGD_degree': eval_SGD_degree,
        'SGD_cc': eval_SGD_cc
    }

    result = []
    for m in metrics:
        if m in ['SGD_degree','SGD_cc']:
            r = method[m](data, **kwargs)
        else:
            r = method[m](data)
        result.append(r)

    if len(result) >= 2:
        return tuple(result)
    else:
        return result[0]