DFE Inference#
Classes for inferring the distribution of fitness effects (DFE) from site-frequency spectra, spanning single-type inference, joint inference across multiple types, and the shared-parameter and covariate machinery that links them.
Classes:
Base inference class for inferring the DFE given one neutral and one selected SFS. |
|
Enabling the sharing of parameters among several Inference objects. |
|
Static utility methods for inference objects. |
|
Inference result. |
|
Class specifying the sharing of params among types. |
|
Class defining a covariate which induces a relationship with one or many parameters. |
- class BaseInference(sfs_neut: Spectra | Spectrum, sfs_sel: Spectra | Spectrum, intervals_del: Tuple[float, float, int] = (-100000000.0, -1e-05, 1000), intervals_ben: Tuple[float, float, int] = (1e-05, 10000.0, 1000), intervals_h: Tuple[float, float, int] = (0.0, 1.0, 21), h_callback: Callable[[float, ndarray], ndarray] = None, integration_mode: Literal['midpoint', 'quad'] = 'midpoint', linearized: bool = True, model: Parametrization | str = 'GammaExpParametrization', seed: int = 0, x0: Dict[str, Dict[str, float]] = {}, bounds: Dict[str, Tuple[float, float]] = {}, scales: Dict[str, Literal['lin', 'log', 'symlog']] = {}, loss_type: Literal['likelihood', 'L2'] = 'likelihood', opts_mle: dict = {}, method_mle: str = 'L-BFGS-B', n_runs: int = 10, fixed_params: Dict[str, Dict[str, float]] = None, do_bootstrap: bool = True, n_bootstraps: int = 100, n_bootstrap_retries: int = 2, parallelize: bool = True, folded: bool = None, discretization: Discretization = None, optimization: Optimization = None, locked: bool = False, include_divergence: bool = True, n_sites_div_neut: float = None, n_sites_div_sel: float = None, **kwargs)[source]#
Bases:
AbstractInferenceBase inference class for inferring the DFE given one neutral and one selected SFS. Note that BaseInference is by default seeded.
Example usage:
import fastdfe as fd sfs_neut = fd.Spectrum([177130, 997, 441, 228, 156, 117, 114, 83, 105, 109, 652]) sfs_sel = fd.Spectrum([797939, 1329, 499, 265, 162, 104, 117, 90, 94, 119, 794]) # create inference object inf = fd.BaseInference( sfs_neut=sfs_neut, sfs_sel=sfs_sel, n_runs=10, n_bootstraps=100, do_bootstrap=True ) # run inference inf.run() # plot discretized DFE inf.plot_discretized() # plot SFS comparison inf.plot_sfs_comparison()
Methods:
Create BaseInference instance.
Get the names of the fixed parameters.
Check that the fixed parameters are valid.
Raise an error if this object is locked.
Run if not run yet.
Perform the DFE inference.
Get callback functions for modelling SFS counts from given parameters.
Evaluate likelihood function at given parameters.
Residual of the modelled and observed SFS.
Perform the parametric bootstrap.
Get linear scales for all parameters.
Get linear bounds for all parameters.
Plot continuous DFE.
Plot mass in each bucket for the MLE DFE.
Plot density of the discretization intervals chosen.
Plot SFS comparison.
Plot neutral and selected SFS.
Plot a bunch of plots.
Get statistics for the discretized DFE.
Visualize the inferred parameters and their confidence intervals.
Visualize the inferred parameters and their confidence intervals.
Visualize the likelihoods of the optimization runs using a scatter plot.
Perform a likelihood ratio test (LRT) with correction for boundary constraints as described in Molenberghs and Verbeke (2007) (https://doi.org/10.1198/000313007X171322).
Perform likelihood ratio test with given more complex model.
Compare the various nested versions of the specified model using likelihood ratio tests.
Plot the p-values of nested likelihoods.
Plot discretized DFE.
Return a frozen DFE object containing the inferred parameters and (if available) the bootstrap samples.
Get discretized DFE.
Return a Spectra object containing the neutral, selected and modelled SFS.
Get alpha, the proportion of beneficial non-synonymous substitutions.
Get omega, the rate of non-synonymous over synonymous substitutions (dN/dS).
Get omega_a, the rate of adaptive non-synonymous over synonymous substitutions (adaptive dN/dS), which equals
alpha * omega.Get the parameters to be included in the bootstraps.
Get the names of the parameters to be included in the bootstraps.
Get the parameters names for the parameters that are optimized.
Get the number of parameters that are optimized.
Create a config object from the inference object.
Load from config object.
Load from config file.
Get summary.
Get initial values.
Load object from file.
Unserialize object.
Save object to file.
Serialize object.
- __init__(sfs_neut: Spectra | Spectrum, sfs_sel: Spectra | Spectrum, intervals_del: Tuple[float, float, int] = (-100000000.0, -1e-05, 1000), intervals_ben: Tuple[float, float, int] = (1e-05, 10000.0, 1000), intervals_h: Tuple[float, float, int] = (0.0, 1.0, 21), h_callback: Callable[[float, ndarray], ndarray] = None, integration_mode: Literal['midpoint', 'quad'] = 'midpoint', linearized: bool = True, model: Parametrization | str = 'GammaExpParametrization', seed: int = 0, x0: Dict[str, Dict[str, float]] = {}, bounds: Dict[str, Tuple[float, float]] = {}, scales: Dict[str, Literal['lin', 'log', 'symlog']] = {}, loss_type: Literal['likelihood', 'L2'] = 'likelihood', opts_mle: dict = {}, method_mle: str = 'L-BFGS-B', n_runs: int = 10, fixed_params: Dict[str, Dict[str, float]] = None, do_bootstrap: bool = True, n_bootstraps: int = 100, n_bootstrap_retries: int = 2, parallelize: bool = True, folded: bool = None, discretization: Discretization = None, optimization: Optimization = None, locked: bool = False, include_divergence: bool = True, n_sites_div_neut: float = None, n_sites_div_sel: float = None, **kwargs)[source]#
Create BaseInference instance.
- Parameters:
sfs_neut¶ (
Spectra|Spectrum) – The neutral SFS. Note that we require monomorphic counts to be specified in order to infer the mutation rate. If aSpectraobject with more than one SFS is provided, theallattribute will be used.sfs_sel¶ (
Spectra|Spectrum) – The selected SFS. Note that we require monomorphic counts to be specified in order to infer the mutation rate. If aSpectraobject with more than one SFS is provided, theallattribute will be used.intervals_del¶ (
Tuple[float,float,int]) –(start, stop, n_interval)for deleterious population-scaled selection coefficients. The intervals will be log10-spaced. Decreasing the number of intervals to100provides nearly identical results while increasing speed, especially when precomputing across dominance coefficients.intervals_ben¶ (
Tuple[float,float,int]) – Same asintervals_delbut for positive selection coefficients. Decreasing the number of intervals to100provides nearly identical results while increasing speed, especially when precomputing across dominance coefficients.intervals_h¶ (
Tuple[float,float,int]) –(start, stop, n_points)for the linearly spaced grid of dominance coefficients. This is only used when inferring dominance coefficients. Values ofhbetween the grid points will be interpolated linearly.h_callback¶ (
Callable[[float,ndarray],ndarray]) – A function mapping the scalar parameterhand the array of selection coefficientsSto dominance coefficients of the same shape, allowing models wherehdepends onS. The default islambda h, S: np.full_like(S, h), keepinghconstant. Expected allele counts for a given dominance value are obtained by linear interpolation between precomputed values inintervals_h. The inferred parameter is still namedh, even if transformed byh_callback, and its bounds, scales, and initial values can be set viabounds,scales, andx0. The fitness of heterozygotes and mutation homozygotes is defined as1 + 2hsand1 + 2s, respectively.integration_mode¶ (
Literal['midpoint','quad']) – Integration mode when computing expected SFS under semidominance.quadis not recommended.linearized¶ (
bool) – Whether to discretize and cache the linearized integral mapping DFE to SFS or usescipy.integrate.quad()in each call.Falseis slow, intended for validation only and deprecated.model¶ (
Parametrization|str) – DFE parametrization, given as aParametrizationinstance or as the name of its class.seed¶ (
int) – Seed for the random number generator. UseNonefor no seed.x0¶ (
Dict[str,Dict[str,float]]) – Dictionary of initial values in the form{'all': {param: value}}bounds¶ (
Dict[str,Tuple[float,float]]) – Bounds for the optimization in the form{param: (lower, upper)}scales¶ (
Dict[str,Literal['lin','log','symlog']]) – Scales for the optimization in the form{param: scale}loss_type¶ (
Literal['likelihood','L2']) – Type of loss function to use for optimization.method_mle¶ (
str) – Method to use for optimization, which needs to support bounds. Seescipy.optimize.minimize()for available methods.n_runs¶ (
int) – Number of independent optimization runs out of which the best one is chosen. The first run will use the initial values if specified. Consider increasing this number if the optimization does not produce good results.fixed_params¶ (
Dict[str,Dict[str,float]]) – Parameters kept constant during optimization, given as{'all': {param: value}}. By default, the ancestral misidentification rateepsis fixed to zero,hto 0.5, and the DFE parameters are fixed so that only the deleterious component of the DFE is estimated.n_bootstrap_retries¶ (
int) – Number of optimization runs for each bootstrap sample. This parameter previously defined the number of retries per bootstrap sample when subsequent runs failed, but now it defines the total number of runs per bootstrap sample, taking the most likely one.folded¶ (
bool) – Whether the SFS are folded. If not specified, the SFS will be folded if both of the given SFS appear to be folded.discretization¶ (
Discretization) – Discretization instance. Mainly intended for internal use.optimization¶ (
Optimization) – Optimization instance. Mainly intended for internal use.locked¶ (
bool) – Whether running inferences from this object is disallowed. Used internally.include_divergence¶ (
bool) – Whether to include divergence counts (fixed differences to an outgroup) in the likelihood, likepolydfe. This only takes effect when both spectra carry a divergence target size, i.e. whenn_sites_div_neutandn_sites_div_selare given; otherwise inference is based on polymorphism alone. The last entry of each SFS is the divergence count. When active, the divergence scale (the unknown outgroup branch length) is derived in closed form from the neutral divergence, and \(\alpha\) can be estimated from the observed divergence (seeget_alpha()). Defaults toTrue. Note that, unlikepolydfe, we do not model misattributed polymorphism (polymorphic sites that appear fixed in a finite sample); we tried this but it did not improve agreement withpolydfeor recovery of the true \(\alpha\) inSLiMsimulations.n_sites_div_neut¶ (
float) – Number of mutational target sites over which neutral divergence was counted. polyDFE allows this to differ from the polymorphism target size. Specifying it (together withn_sites_div_sel) opts the neutral divergence into the likelihood.n_sites_div_sel¶ (
float) – Number of mutational target sites over which selected divergence was counted. Specifying it (together withn_sites_div_neut) opts the selected divergence into the likelihood.kwargs¶ – Additional keyword arguments which are ignored.
- sfs_neut: Spectrum#
Neutral SFS
- sfs_sel: Spectrum#
Selected SFS
-
n_sites_div_neut:
Optional[float] = None# Number of mutational target sites over which neutral divergence was counted. Owned by the inference (not the spectra); specifying both this and
n_sites_div_selopts divergence into the likelihood.
-
n_sites_div_sel:
Optional[float] = None# Number of mutational target sites over which selected divergence was counted.
- model: Parametrization#
The DFE parametrization
- folded: bool#
Whether the SFS are folded
- n: int#
Sample size
- fixed_params: Dict[str, Dict[str, float]]#
Fixed parameters
- discretization: Discretization#
Discretization instance
- theta: float#
Estimate of theta from neutral SFS
- params_mle: Optional[Dict[str, float]]#
MLE estimates of the initial optimization
- sfs_mle: Optional[Spectrum]#
Modelled MLE SFS
- likelihood: Optional[float]#
Likelihood of the MLE, this value may be updated after bootstrapping
- likelihoods: Optional[List[float]]#
Likelihoods of the different ML runs, controlled by
n_runs
- n_runs: int#
Number of MLE runs to perform
- result: Optional['scipy.optimize.OptimizeResult']#
Numerical optimization result
- do_bootstrap: bool#
Whether to do bootstrapping
- n_bootstraps: int#
Number of bootstraps
- n_bootstrap_retries: int#
Number of optimization runs for each bootstrap sample
- parallelize: bool#
Whether to parallelize computations
- scales: Dict[str, Literal['lin', 'log', 'symlog']]#
Parameter scales
- bounds: Dict[str, Tuple[float, float]]#
Parameter bounds
- optimization: Optimization#
Optimization instance
- x0: Dict[str, Dict[str, float]]#
Initial values
- bootstraps: Optional[pd.DataFrame]#
Bootstrapped MLE parameter estimates
- bootstrap_results: Optional[List['scipy.optimize.OptimizeResult']]#
Bootstrap optimization results
- L2_residual: Optional[float]#
L2 norm of differential between modelled and observed SFS
- seed: int | None#
Random number generator seed
- rng: np.random.Generator#
Random generator instance
- execution_time: float#
Total execution time in seconds
- locked: bool#
Whether running inferences from this object is disallowed
- run(do_bootstrap: bool = None, pbar: bool = None, **kwargs) Spectrum[source]#
Perform the DFE inference.
- get_counts() dict[source]#
Get callback functions for modelling SFS counts from given parameters.
- Returns:
Callback functions for modelling SFS counts for each type.
- evaluate_likelihood(params: Dict[str, Dict[str, float]]) float[source]#
Evaluate likelihood function at given parameters.
- bootstrap(n_samples: int = None, parallelize: bool = None, update_likelihood: bool = False, n_retries: int = None, pbar: bool = True, pbar_title: str = 'Bootstrapping') DataFrame[source]#
Perform the parametric bootstrap. Replicates are obtained by resampling the neutral and selected SFS from independent Poisson distributions with the observed counts as means.
- Parameters:
n_samples¶ (
int) – Number of bootstrap samples. Defaults ton_bootstraps.parallelize¶ (
bool) – Whether to parallelize the bootstrap. Defaults toparallelize.update_likelihood¶ (
bool) – Whether to update the likelihood to be the mean of the bootstrap samples and the original likelihood. This is not statistically valid but can stabilize maximum likelihood estimate subject to optimization noise.n_retries¶ (
int) – Number of optimization runs for each bootstrap sample. Defaults ton_bootstrap_retries.
- Returns:
Dataframe with bootstrap results.
- get_scales_linear() Dict[str, Literal['lin']][source]#
Get linear scales for all parameters.
- Returns:
Dictionary of scales.
- get_bounds_linear() Dict[str, Tuple[float, float]][source]#
Get linear bounds for all parameters.
- Returns:
Dictionary of bounds.
- plot_continuous(file: str = None, show: bool = True, intervals: ndarray = None, confidence_intervals: bool = True, ci_level: float = 0.05, bootstrap_type: Literal['percentile', 'bca'] = 'percentile', scale_density: bool = False, scale: Literal['lin', 'log', 'symlog'] = 'lin', title: str = 'DFE', kwargs_legend: dict = {'prop': {'size': 8}}, ax: Axes = None) Axes[source]#
Plot continuous DFE. The special constants np.inf and -np.inf are also valid interval bounds. By default, the PDF is plotted as is. Due to the logarithmic scale on the x-axis, we may get a wrong intuition on how the mass is distributed, however. To get a better intuition, we can optionally scale the density by the x-axis interval size using
scale_density = True. This has the disadvantage that the density now changes for x, so that even a constant density will look warped.- Parameters:
bootstrap_type¶ (
Literal['percentile','bca']) – Type of bootstrap to use for confidence intervals.ci_level¶ (
float) – Tail probability per side, so 0.05 yields a 90% interval.confidence_intervals¶ (
bool) – Whether to plot confidence intervals.scale_density¶ (
bool) – Whether to scale the density by the x-axis interval sizeintervals¶ (
ndarray) – Array of interval boundaries yieldingintervals.shape[0] - 1bins.kwargs_legend¶ (
dict) – Keyword arguments passed toAxes.legend(). Only for Python visualization backend.ax¶ (
Axes) – Axes to plot on. Only for Python visualization backend.
- Returns:
Axes object
- plot_bucket_sizes(file: str = None, show: bool = True, title: str = 'bucket sizes', ax: Axes = None) Axes[source]#
Plot mass in each bucket for the MLE DFE. This can be used to check if the interval bounds and spacing are chosen appropriately.
- plot_interval_density(file: str = None, show: bool = True, intervals: Sequence = array([-inf, -100., -10., -1., 0., 1., inf]), interval_labels: List[str] = None, color: str = 'C0', ax: Axes = None) Axes[source]#
Plot density of the discretization intervals chosen. Note that although this plot looks similar, this is not the DFE!
- plot_sfs_comparison(sfs_types: List[Literal['modelled', 'observed', 'selected', 'neutral']] = ['observed', 'modelled'], labels: List[str] = None, colors: List[str] = None, file: str = None, show: bool = True, ax: Axes = None, title: str = 'SFS comparison', use_subplots: bool = False, show_monomorphic: bool = False, kwargs_legend: dict = {'prop': {'size': 8}}) Axes[source]#
Plot SFS comparison.
- Parameters:
colors¶ (
List[str]) – Colors for the SFS. Only for Python visualization backend.sfs_types¶ (
List[Literal['modelled','observed','selected','neutral']]) – Types of SFS to plot.ax¶ (
Axes) – Axes to plot on. Only for Python visualization backend and ifuse_subplotsisFalse.use_subplots¶ (
bool) – Whether to use subplots. Only for Python visualization backend.show_monomorphic¶ (
bool) – Whether to show monomorphic countskwargs_legend¶ (
dict) – Keyword arguments passed toAxes.legend(). Only for Python visualization backend.
- Returns:
Axes object
- plot_observed_sfs(labels: List[str] = None, file: str = None, show: bool = True, ax: Axes = None) Axes[source]#
Plot neutral and selected SFS.
- get_stats_discretized(ci_level: float = 0.05, intervals: ndarray = array([-inf, -100., -10., -1., 0., 1., inf]), bootstrap_type: Literal['percentile', 'bca'] = 'percentile', point_estimate: Literal['original', 'mean', 'median'] = 'mean') Tuple[ndarray, ndarray, ndarray][source]#
Get statistics for the discretized DFE.
- Parameters:
ci_level¶ (
float) – Tail probability per side, so 0.05 yields a 90% interval.intervals¶ (
ndarray) – Array of interval boundaries yieldingintervals.shape[0] - 1bins.bootstrap_type¶ (
Literal['percentile','bca']) – Type of bootstrap to use.point_estimate¶ (
Literal['original','mean','median']) – Whether to use ‘original’ MLE values, ‘mean’ or ‘median’ of bootstraps as point estimate.
- Returns:
Center values, errors around center, and confidence intervals.
- Raises:
ValueError – If the inference has no bootstraps.
- plot_inferred_parameters(confidence_intervals: bool = True, ci_level: float = 0.05, bootstrap_type: Literal['percentile', 'bca'] = 'percentile', file: str = None, show: bool = True, title: str = 'parameter estimates', scale: Literal['lin', 'log', 'symlog'] = 'log', ax: Axes = None, kwargs_legend: dict = {'loc': 'upper right', 'prop': {'size': 8}}, point_estimate: Literal['original', 'mean', 'median'] = 'mean', **kwargs) Axes[source]#
Visualize the inferred parameters and their confidence intervals.
- Parameters:
point_estimate¶ (
Literal['original','mean','median']) – Whether to use ‘original’ MLE values, ‘mean’ or ‘median’ of bootstraps as point estimate.scale¶ (
Literal['lin','log','symlog']) – y-scale of the plot.confidence_intervals¶ (
bool) – Whether to show confidence intervals.bootstrap_type¶ (
Literal['percentile','bca']) – Type of bootstrap to use.ci_level¶ (
float) – Tail probability per side, so 0.05 yields a 90% interval.ax¶ (
Axes) – Axes to plot on. Only for Python visualization backend.kwargs_legend¶ (
dict) – Keyword arguments passed toAxes.legend(). Only for Python visualization backend.
- Returns:
Axes object
- plot_inferred_parameters_boxplot(file: str = None, show: bool = True, title: str = 'parameter estimates', **kwargs) Axes[source]#
Visualize the inferred parameters and their confidence intervals.
- plot_likelihoods(file: str = None, show: bool = True, title: str = 'likelihoods', scale: Literal['lin', 'log'] = 'lin', ax: Axes = None, ylabel: str = 'lnl', **kwargs) Axes[source]#
Visualize the likelihoods of the optimization runs using a scatter plot.
- Parameters:
- Returns:
Axes object
- Raises:
ValueError – If the inference holds no likelihoods of its optimization runs.
- lrt(ll_simple: float, ll_complex: float, df: int, m: int = 0) float[source]#
Perform a likelihood ratio test (LRT) with correction for boundary constraints as described in Molenberghs and Verbeke (2007) (https://doi.org/10.1198/000313007X171322).
- Parameters:
- Returns:
p-value
- Raises:
ValueError – If the loss function of the inference is not the likelihood.
- compare_nested(complex: BaseInference) float | None[source]#
Perform likelihood ratio test with given more complex model. The given model’s fixed parameters need to be a proper subset of this model’s fixed parameters. Low p-values indicate that the more complex model provides a significantly better fit.
- Parameters:
complex¶ (
BaseInference) – More complex model.- Returns:
p-value.
- Raises:
ValueError – If the inference objects are not nested.
- compare_nested_models(do_bootstrap: bool = False) Tuple[ndarray, Dict[str, BaseInference]][source]#
Compare the various nested versions of the specified model using likelihood ratio tests. We check for inclusion of ancestral misidentification and beneficial mutations.
- plot_nested_models(file: str = None, show: bool = True, remove_empty: bool = False, transpose: bool = False, cmap: str = None, title: str = 'nested model comparison', ax: Axes = None, do_bootstrap: bool = False) Axes[source]#
Plot the p-values of nested likelihoods.
- plot_discretized(file: str = None, show: bool = True, intervals: ndarray = array([-inf, -100., -10., -1., 0., 1., inf]), confidence_intervals: bool = True, ci_level: float = 0.05, bootstrap_type: Literal['percentile', 'bca'] = 'percentile', point_estimate: Literal['original', 'mean', 'median'] = 'mean', title: str = 'discretized DFE', ax: plt.Axes = None, kwargs_legend: dict = {'prop': {'size': 8}}) plt.Axes#
Plot discretized DFE.
- Parameters:
intervals¶ (
ndarray) – Array of interval boundaries over(-inf, inf)yieldingintervals.shape[0] - 1bars.confidence_intervals¶ (
bool) – Whether to plot confidence intervalsci_level¶ (
float) – Tail probability per side, so 0.05 yields a 90% interval.bootstrap_type¶ (
Literal['percentile','bca']) – Type of bootstrappoint_estimate¶ (
Literal['original','mean','median']) – Whether to use ‘original’ MLE values, ‘mean’ or ‘median’ of bootstraps as point estimate.ax¶ (
Axes) – Axes to plot on. Only for Python visualization backend.kwargs_legend¶ (
dict) – Keyword arguments passed toAxes.legend(). Only for Python visualization backend.
- Returns:
Axes
- get_dfe() DFE[source]#
Return a frozen DFE object containing the inferred parameters and (if available) the bootstrap samples.
- Returns:
DFE object
- get_discretized(intervals: ndarray = array([-inf, -100., -10., -1., 0., 1., inf]), confidence_intervals: bool = True, ci_level: float = 0.05, bootstrap_type: Literal['percentile', 'bca'] = 'percentile', point_estimate: Literal['original', 'mean', 'median'] = 'mean') Tuple[ndarray, ndarray | None]#
Get discretized DFE.
- Parameters:
intervals¶ (
ndarray) – Array of interval boundaries over(-inf, inf)yieldingintervals.shape[0] - 1bins.confidence_intervals¶ (
bool) – Whether to return confidence intervalsci_level¶ (
float) – Tail probability per side, so 0.05 yields a 90% interval.bootstrap_type¶ (
Literal['percentile','bca']) – Type of bootstrappoint_estimate¶ (
Literal['original','mean','median']) – Whether to use ‘original’ MLE values, ‘mean’ or ‘median’ of bootstraps as point estimate.
- Returns:
Array of values and array of deviations
- get_spectra() Spectra[source]#
Return a Spectra object containing the neutral, selected and modelled SFS.
- Returns:
Spectra object with types
neutral,selected, andmodelled, folded if the inference is.
- get_alpha(params: dict = None, use_divergence: bool = None) float[source]#
Get alpha, the proportion of beneficial non-synonymous substitutions.
- Parameters:
params¶ (
dict) – Parameters to use for calculation, as inparams_mle. The dominance coefficienthis taken to be 0.5 if absent.use_divergence¶ (
bool) – Whether to estimate alpha from observed divergence counts using polyDFE’s McDonald-Kreitman style estimator, or from the DFE alone. WhenNone(the default), this follows the inference mode (whether divergence counts are used in the likelihood). PassingFalseexplicitly always yields the polymorphism-only estimate, which is useful for comparison.
- Returns:
Estimate of alpha.
- Raises:
ValueError – If
use_divergenceisTruewithout divergence target sizes or divergence counts.
- get_omega(params: dict = None, use_divergence: bool = None) float[source]#
Get omega, the rate of non-synonymous over synonymous substitutions (dN/dS).
- Parameters:
params¶ (
dict) – Parameters to use for calculation, as inparams_mle. The dominance coefficienthis taken to be 0.5 if absent.use_divergence¶ (
bool) – Whether to estimate omega from observed divergence counts using polyDFE’s McDonald-Kreitman style estimator, or from the DFE alone. WhenNone(the default), this follows the inference mode (whether divergence counts are used in the likelihood). PassingFalseexplicitly always yields the polymorphism-only estimate, which is useful for comparison.
- Returns:
Estimate of omega.
- Raises:
ValueError – If
use_divergenceisTruewithout divergence target sizes or divergence counts.
- get_omega_a(params: dict = None, use_divergence: bool = None) float[source]#
Get omega_a, the rate of adaptive non-synonymous over synonymous substitutions (adaptive dN/dS), which equals
alpha * omega.- Parameters:
params¶ (
dict) – Parameters to use for calculation, as inparams_mle. The dominance coefficienthis taken to be 0.5 if absent.use_divergence¶ (
bool) – Whether to estimate omega_a from observed divergence counts using polyDFE’s McDonald-Kreitman style estimator, or from the DFE alone. WhenNone(the default), this follows the inference mode (whether divergence counts are used in the likelihood). PassingFalseexplicitly always yields the polymorphism-only estimate, which is useful for comparison.
- Returns:
Estimate of omega_a.
- Raises:
ValueError – If
use_divergenceisTruewithout divergence target sizes or divergence counts.
- property omega: float#
Cache omega, the rate of non-synonymous over synonymous substitutions (dN/dS).
- property omega_a: float#
Cache omega_a, the rate of adaptive non-synonymous over synonymous substitutions (adaptive dN/dS).
- get_bootstrap_params() Dict[str, float][source]#
Get the parameters to be included in the bootstraps.
- Returns:
Parameters to be included in the bootstraps.
- get_bootstrap_param_names() List[str][source]#
Get the names of the parameters to be included in the bootstraps.
- Returns:
Names of the parameters to be included in the bootstraps.
- get_optimized_param_names() List[str][source]#
Get the parameters names for the parameters that are optimized.
- Returns:
List of parameter names.
- get_n_optimized() int[source]#
Get the number of parameters that are optimized.
- Returns:
Number of parameters that are optimized.
- create_config() Config[source]#
Create a config object from the inference object.
- Returns:
Config object.
- get_summary() InferenceResult[source]#
Get summary.
- Returns:
Inference results.
- class JointInference(sfs_neut: Spectra, sfs_sel: Spectra, include_divergence: bool = True, n_sites_div_neut: Dict[str, float] = None, n_sites_div_sel: Dict[str, float] = None, intervals_del: Tuple[float, float, int] = (-100000000.0, -1e-05, 1000), intervals_ben: Tuple[float, float, int] = (1e-05, 10000.0, 1000), intervals_h: Tuple[float, float, int] = (0.0, 1.0, 21), h_callback: Callable[[float, ndarray], ndarray] = None, integration_mode: Literal['midpoint', 'quad'] = 'midpoint', linearized: bool = True, model: Parametrization | str = 'GammaExpParametrization', seed: int = 0, x0: Dict[str, Dict[str, float]] = {}, bounds: Dict[str, Tuple[float, float]] = {}, scales: Dict[str, Literal['lin', 'log', 'symlog']] = {}, loss_type: Literal['likelihood', 'L2'] = 'likelihood', opts_mle: dict = {}, method_mle: str = 'L-BFGS-B', n_runs: int = 10, fixed_params: Dict[str, Dict[str, float]] = None, shared_params: List[SharedParams] = [], covariates: List[Covariate] = [], do_bootstrap: bool = True, n_bootstraps: int = 100, n_bootstrap_retries: int = 2, parallelize: bool = True, folded: bool = None, discretization: Discretization = None, optimization: Optimization = None, locked: bool = False, **kwargs)[source]#
Bases:
BaseInferenceEnabling the sharing of parameters among several Inference objects.
Example usage:
import fastdfe as fd # neutral SFS for two types sfs_neut = fd.Spectra(dict( pendula=[177130, 997, 441, 228, 156, 117, 114, 83, 105, 109, 652], pubescens=[172528, 3612, 1359, 790, 584, 427, 325, 234, 166, 76, 31] )) # selected SFS for two types sfs_sel = fd.Spectra(dict( pendula=[797939, 1329, 499, 265, 162, 104, 117, 90, 94, 119, 794], pubescens=[791106, 5326, 1741, 1005, 756, 546, 416, 294, 177, 104, 41] )) # create inference object inf = fd.JointInference( sfs_neut=sfs_neut, sfs_sel=sfs_sel, do_bootstrap=True ) # run inference inf.run()
Methods:
Create instance.
Check if the shared parameters were specified correctly.
Check that no shared parameters are fixed and raise an error otherwise.
Get the names of the shared parameters.
Load from config object.
Run inference.
Get marginal inference without 'all' type.
Check if joint inference makes sense.
Evaluate likelihood function at given parameters.
Get callback functions for modelling SFS counts from given parameters.
Run joint inference without covariates.
Create a config object from the inference object.
Perform likelihood ratio test against joint inference without covariates.
Perform the parametric bootstrap both for the marginal and joint inferences.
Bootstrap if not done yet.
Get initial values for joint inference.
Compare likelihood of joint inference with product of marginal likelihoods.
Get all inference objects as dictionary.
Plot discretized DFE comparing the different types.
Plot SFS comparison.
Plot the continuous DFE of the different types.
Plot the inferred parameters of the different types.
Plot the bootstrap distributions of the inferred parameters of the different types as boxplots.
Plot the covariate given by the index.
Get discretized DFEs.
Get statistics for the discretized DFEs.
Not available for a joint inference, whose discretization is shared by its inferences.
Get bootstrap parameters.
Serialize object.
Return the inferred DFE of each type.
Return the neutral, selected and modelled SFS of each type.
Get the summary of each type.
Check that the fixed parameters are valid.
Perform likelihood ratio test with given more complex model.
Compare the various nested versions of the specified model using likelihood ratio tests.
Load from config file.
Load object from file.
Unserialize object.
Get the names of the parameters to be included in the bootstraps.
Get linear bounds for all parameters.
Get the names of the fixed parameters.
Get the number of parameters that are optimized.
Get the parameters names for the parameters that are optimized.
Get linear scales for all parameters.
Perform a likelihood ratio test (LRT) with correction for boundary constraints as described in Molenberghs and Verbeke (2007) (https://doi.org/10.1198/000313007X171322).
Plot a bunch of plots.
Plot density of the discretization intervals chosen.
Visualize the likelihoods of the optimization runs using a scatter plot.
Plot the p-values of nested likelihoods.
Plot neutral and selected SFS.
Raise an error if this object is locked.
Run if not run yet.
Save object to file.
Residual of joint inference.
Get alpha, the proportion of beneficial non-synonymous substitutions, for each type.
Get omega, the rate of non-synonymous over synonymous substitutions (dN/dS), for each type.
Get omega_a, the rate of adaptive non-synonymous over synonymous substitutions (adaptive dN/dS), for each type.
- __init__(sfs_neut: Spectra, sfs_sel: Spectra, include_divergence: bool = True, n_sites_div_neut: Dict[str, float] = None, n_sites_div_sel: Dict[str, float] = None, intervals_del: Tuple[float, float, int] = (-100000000.0, -1e-05, 1000), intervals_ben: Tuple[float, float, int] = (1e-05, 10000.0, 1000), intervals_h: Tuple[float, float, int] = (0.0, 1.0, 21), h_callback: Callable[[float, ndarray], ndarray] = None, integration_mode: Literal['midpoint', 'quad'] = 'midpoint', linearized: bool = True, model: Parametrization | str = 'GammaExpParametrization', seed: int = 0, x0: Dict[str, Dict[str, float]] = {}, bounds: Dict[str, Tuple[float, float]] = {}, scales: Dict[str, Literal['lin', 'log', 'symlog']] = {}, loss_type: Literal['likelihood', 'L2'] = 'likelihood', opts_mle: dict = {}, method_mle: str = 'L-BFGS-B', n_runs: int = 10, fixed_params: Dict[str, Dict[str, float]] = None, shared_params: List[SharedParams] = [], covariates: List[Covariate] = [], do_bootstrap: bool = True, n_bootstraps: int = 100, n_bootstrap_retries: int = 2, parallelize: bool = True, folded: bool = None, discretization: Discretization = None, optimization: Optimization = None, locked: bool = False, **kwargs)[source]#
Create instance.
- Parameters:
sfs_neut¶ (
Spectra) – Neutral SFS. Note that we require monomorphic counts to be specified in order to infer the mutation rate.sfs_sel¶ (
Spectra) – Selected SFS. Note that we require monomorphic counts to be specified in order to infer the mutation rate.include_divergence¶ (
bool) – Whether to include divergence counts in the likelihood. Only takes effect when the divergence target sizes are given vian_sites_div_neutandn_sites_div_sel; seeBaseInferencefor details.n_sites_div_neut¶ (
Dict[str,float]) – Number of mutational target sites over which neutral divergence was counted, keyed by type. Specifying it (withn_sites_div_sel) opts neutral divergence into the likelihood.n_sites_div_sel¶ (
Dict[str,float]) – Number of mutational target sites over which selected divergence was counted, keyed by type. Specifying it (withn_sites_div_neut) opts selected divergence into the likelihood.intervals_del¶ (
Tuple[float,float,int]) –(start, stop, n_interval)for deleterious population-scaled selection coefficients. The intervals will be log10-spaced. Decreasing the number of intervals to100provides nearly identical results while increasing speed, especially when precomputing across dominance coefficients.intervals_ben¶ (
Tuple[float,float,int]) – Same asintervals_delbut for positive selection coefficients. Decreasing the number of intervals to100provides nearly identical results while increasing speed, especially when precomputing across dominance coefficients.intervals_h¶ (
Tuple[float,float,int]) –(start, stop, n_points)for the linearly spaced grid of dominance coefficients. This is only used when inferring dominance coefficients. Values ofhbetween the grid points will be interpolated linearly.h_callback¶ (
Callable[[float,ndarray],ndarray]) – A function mapping the scalar parameterhand the array of selection coefficientsSto dominance coefficients of the same shape, allowing models wherehdepends onS. The default islambda h, S: np.full_like(S, h), keepinghconstant. Expected allele counts for a given dominance value are obtained by linear interpolation between precomputed values inintervals_h. The inferred parameter is still namedh, even if transformed byh_callback, and its bounds, scales, and initial values can be set viabounds,scales, andx0. The fitness of heterozygotes and mutation homozygotes is defined as1 + 2hsand1 + 2s, respectively.integration_mode¶ (
Literal['midpoint','quad']) – Integration mode when computing expected SFS under semidominance.quadis not recommended.linearized¶ (
bool) – Whether to discretize and cache the linearized integral mapping DFE to SFS or usescipy.integrate.quad()in each call.Falseis slow, intended for validation only and deprecated.model¶ (
Parametrization|str) – DFE parametrizationx0¶ (
Dict[str,Dict[str,float]]) – Dictionary of initial values in the form{type: {param: value}}. The entryallholds the initial values of the inference on the spectra pooled over all types. The marginal inference of a type starts from the pooled estimates, with the parameters given under that type taking precedence. The joint inference starts from the marginal estimates.bounds¶ (
Dict[str,Tuple[float,float]]) – Bounds for the optimization in the form{param: (lower, upper)}scales¶ (
Dict[str,Literal['lin','log','symlog']]) – Scales for the optimization in the form{param: scale}method_mle¶ (
str) – Method to use for optimization, which needs to support bounds. Seescipy.optimize.minimize()for available methods.n_runs¶ (
int) – Number of independent optimization runs out of which the best one is chosen. The first run will use the initial values if specified. Consider increasing this number if the optimization does not produce good results.fixed_params¶ (
Dict[str,Dict[str,float]]) – Parameters kept constant during optimization, given as{type: {param: value}}. By default, the ancestral misidentification rateepsis fixed to zero,hto 0.5, and the DFE parameters are fixed so that only the deleterious component of the DFE is estimated.shared_params¶ (
List[SharedParams]) – List of shared parameterscovariates¶ (
List[Covariate]) – List of covariates. Parameters with a covariate are shared across all types.n_bootstrap_retries¶ (
int) – Number of optimization runs for each bootstrap sample. This parameter previously defined the number of retries per bootstrap sample when subsequent runs failed, but now it defines the total number of runs per bootstrap sample, taking the most likely one.folded¶ (
bool) – Whether the SFS are folded. If not specified, the SFS will be folded if all of the given SFS appear to be folded.discretization¶ (
Discretization) – Discretization instance. Mainly intended for internal use.optimization¶ (
Optimization) – Optimization instance. Mainly intended for internal use.locked¶ (
bool) – Whether running the inference from this object is disallowed. Used internally.kwargs¶ – Additional keyword arguments which are ignored.
- Raises:
ValueError – If
x0contains a key that is neitherallnor a type, if a type is namedallor contains a dot or colon, or if covariates are given for a single type.
- types: List[str]#
SFS types
- covariates: Dict[str, Covariate]#
Covariates indexed by parameter name
- params_mle_raw: Optional[Dict[str, Dict[str, float]]]#
original MLE parameters before adding covariates and unpacked shared
- fixed_params: Dict[str, Dict[str, float]]#
fixed parameters with expanded ‘all’ type
Shared parameters with expanded ‘all’ type, all parameters being those that are not fixed
- discretization: Discretization#
Discretization instance
- marginal_inferences: Dict[str, BaseInference]#
Dictionary of marginal inferences indexed by type
- include_divergence: bool = True#
Whether divergence counts were requested
- scales: Dict[str, Literal['lin', 'log', 'symlog']]#
parameter scales
- bounds: Dict[str, Tuple[float, float]]#
parameter bounds
- optimization: Optimization#
Joint optimization instance
- joint_inferences: Dict[str, BaseInference]#
Joint inference object indexed by type
Check if the shared parameters were specified correctly.
Check that no shared parameters are fixed and raise an error otherwise.
Get the names of the shared parameters.
- Returns:
Names of the shared parameters.
- classmethod from_config(config: Config) JointInference[source]#
Load from config object. Unlike
BaseInference.from_config(), the per-type divergence target sizes are passed through unchanged (JointInference expects them keyed by type).
- marginals_without_all() Dict[str, BaseInference][source]#
Get marginal inference without ‘all’ type.
- Returns:
Dict of marginal inferences indexed by type.
- joint_inference_makes_sense() bool[source]#
Check if joint inference makes sense.
- Returns:
Whether joint inference makes sense.
- evaluate_likelihood(params: Dict[str, Dict[str, float]]) float[source]#
Evaluate likelihood function at given parameters.
- Parameters:
params¶ (
Dict[str,Dict[str,float]]) – Parameters indexed by type and parameter name. Parameters given for each type, as inparams_mle, include the effect of the covariates. Parameters given with the shared ones packed, as inparams_mle_raw, have the covariates applied to them.- Returns:
The log-likelihood, or the negative L2 residual if the loss type is
L2.
- get_counts() dict[source]#
Get callback functions for modelling SFS counts from given parameters.
- Returns:
Dict of callback functions indexed by type.
- run_joint_without_covariates(do_bootstrap: bool = True) JointInference[source]#
Run joint inference without covariates. Note that the result of this function is cached.
- create_config() Config[source]#
Create a config object from the inference object.
- Returns:
Config object.
- perform_lrt_covariates(do_bootstrap: bool = False) float[source]#
Perform likelihood ratio test against joint inference without covariates. In the simple model we share parameters across types. Low p-values indicate that including covariates significantly improves the model fit.
To access the JointInference object without covariates, you can call
run_joint_without_covariates(), which is cached.- Parameters:
- Returns:
Likelihood ratio test p-value.
- Raises:
ValueError – If no covariates were specified.
- bootstrap(n_samples: int = None, parallelize: bool = None, update_likelihood: bool = False, n_retries: int = None, pbar: bool = True, pbar_title: str = 'Bootstrapping joint inference') DataFrame | None[source]#
Perform the parametric bootstrap both for the marginal and joint inferences. Replicates are obtained by resampling the neutral and selected SFS of each type from independent Poisson distributions with the observed counts as means.
- Parameters:
n_samples¶ (
int) – Number of bootstrap samples. Defaults ton_bootstraps.parallelize¶ (
bool) – Whether to parallelize the bootstrap. Defaults toparallelize.update_likelihood¶ (
bool) – Whether to update the likelihood to be the mean of the bootstrap samples and the original likelihood. This is not statistically valid but can stabilize maximum likelihood estimate subject optimization noise.n_retries¶ (
int) – Number of optimization runs for each bootstrap sample. Defaults ton_bootstrap_retries.
- Returns:
DataFrame with bootstrap samples
- get_x0() Dict[str, Dict[str, float]][source]#
Get initial values for joint inference.
- Returns:
Dictionary of initial values indexed by type
Compare likelihood of joint inference with product of marginal likelihoods. This provides information about the goodness of fit achieved by the parameter sharing. Low p-values indicate that parameter sharing is not justified, i.e., that the marginal inferences provide a significantly better fit to the data. Note that it is more difficult to properly optimize the joint likelihood, which inflates the test statistic, i.e., the reported p-value may be smaller than what it should be.
- get_inferences(types: List[str] = None, labels: List[str] = None, show_marginals: bool = True) Dict[str, BaseInference][source]#
Get all inference objects as dictionary.
- Parameters:
- Returns:
Dictionary of base inference objects indexed by type and joint vs marginal subtypes
- Raises:
ValueError – If the number of labels does not match the number of inferences.
- plot_discretized(file: str = None, show: bool = True, intervals: ndarray = array([-inf, -100., -10., -1., 0., 1., inf]), show_marginals: bool = True, confidence_intervals: bool = True, ci_level: float = 0.05, bootstrap_type: Literal['percentile', 'bca'] = 'percentile', point_estimate: Literal['original', 'mean', 'median'] = 'original', title: str = 'discretized DFE comparison', labels: List[str] = None, kwargs_legend: dict = {'prop': {'size': 8}}, ax: Axes = None) Axes[source]#
Plot discretized DFE comparing the different types.
- Parameters:
labels¶ (
List[str]) – Labels, one per plotted inference, in the order ofJointInference.get_inferences().intervals¶ (
ndarray) – Array of interval boundaries over(-inf, inf)yieldingintervals.shape[0] - 1bars.show_marginals¶ (
bool) – Whether to also show marginal inferencesbootstrap_type¶ (
Literal['percentile','bca']) – Type of bootstrappoint_estimate¶ (
Literal['original','mean','median']) – Whether to use ‘original’ MLE values, ‘mean’ or ‘median’ of bootstraps as point estimate.ci_level¶ (
float) – Tail probability per side, so 0.05 yields a 90% interval.confidence_intervals¶ (
bool) – Whether to plot confidence intervalsax¶ (
Axes) – Axes to plot on. Only for Python visualization backend.kwargs_legend¶ (
dict) – Keyword arguments passed toAxes.legend(). Only for Python visualization backend.
- Returns:
Axes object
- plot_sfs_comparison(sfs_types: List[Literal['modelled', 'observed', 'selected', 'neutral']] = ['observed', 'modelled'], types: List[str] = None, labels: List[str] = None, colors: List[str] = None, file: str = None, show: bool = True, ax: Axes = None, title: str = 'SFS comparison', use_subplots: bool = False, show_monomorphic: bool = False, kwargs_legend: dict = {'prop': {'size': 8}}) Axes[source]#
Plot SFS comparison.
- Parameters:
labels¶ (
List[str]) – Labels, one per plotted spectrum, i.e. one per inference and entry ofsfs_types.colors¶ (
List[str]) – Colors, one per plotted spectrum, as forlabels. Only for Python visualization backend.sfs_types¶ (
List[Literal['modelled','observed','selected','neutral']]) – Types of SFS to plotax¶ (
Axes) – Axes to plot on. Only for Python visualization backend and ifuse_subplotsisFalse.use_subplots¶ (
bool) – Whether to use subplots. Only for Python visualization backend.show_monomorphic¶ (
bool) – Whether to show monomorphic countskwargs_legend¶ (
dict) – Keyword arguments passed toAxes.legend(). Only for Python visualization backend.
- Returns:
Axes object
- plot_continuous(file: str = None, show: bool = True, intervals: ndarray = None, confidence_intervals: bool = True, ci_level: float = 0.05, bootstrap_type: Literal['percentile', 'bca'] = 'percentile', title: str = 'DFE comparison', labels: List[str] = None, scale_density: bool = False, scale: Literal['lin', 'log', 'symlog'] = 'lin', kwargs_legend: dict = {'prop': {'size': 8}}, ax: Axes = None) Axes[source]#
Plot the continuous DFE of the different types. The special constants
np.infand-np.infare also valid interval bounds. By default, the PDF is plotted as is. Due to the logarithmic scale on the x-axis, we may get a wrong intuition on how the mass is distributed, however. To get a better intuition, we can optionally scale the density by the x-axis interval size usingscale_density = True. This has the disadvantage that the density now changes for x, so that even a constant density will look warped.- Parameters:
scale_density¶ (
bool) – Whether to scale the density by the x-axis interval sizelabels¶ (
List[str]) – Labels, one per plotted inference, in the order ofJointInference.get_inferences().bootstrap_type¶ (
Literal['percentile','bca']) – Type of bootstrapci_level¶ (
float) – Tail probability per side, so 0.05 yields a 90% interval.confidence_intervals¶ (
bool) – Whether to plot confidence intervalsintervals¶ (
ndarray) – Array of interval boundaries yieldingintervals.shape[0] - 1bins.kwargs_legend¶ (
dict) – Keyword arguments passed toAxes.legend(). Only for Python visualization backend.ax¶ (
Axes) – Axes to plot on. Only for Python visualization backend.
- Returns:
Axes object
- plot_inferred_parameters(file: str = None, confidence_intervals: bool = True, bootstrap_type: Literal['percentile', 'bca'] = 'percentile', ci_level: float = 0.05, show: bool = True, title: str = 'inferred parameters', labels: List[str] = None, ax: Axes = None, scale: Literal['lin', 'log', 'symlog'] = 'log', kwargs_legend: dict = {'loc': 'upper right', 'prop': {'size': 8}}, point_estimate: Literal['original', 'mean', 'median'] = 'mean', **kwargs: List[str]) Axes[source]#
Plot the inferred parameters of the different types.
- Parameters:
labels¶ (
List[str]) – Labels, one per plotted inference, in the order ofJointInference.get_inferences().bootstrap_type¶ (
Literal['percentile','bca']) – Type of bootstrappoint_estimate¶ (
Literal['original','mean','median']) – Whether to use ‘original’ MLE values, ‘mean’ or ‘median’ of bootstraps as point estimate.ci_level¶ (
float) – Tail probability per side, so 0.05 yields a 90% interval.confidence_intervals¶ (
bool) – Whether to plot confidence intervalsax¶ (
Axes) – Axes to plot on. Only for Python visualization backend.kwargs_legend¶ (
dict) – Keyword arguments passed toAxes.legend(). Only for Python visualization backend.
- Returns:
Axes object
- plot_inferred_parameters_boxplot(file: str = None, show: bool = True, title: str = 'inferred parameters', labels: List[str] = None, **kwargs: List[str]) Axes[source]#
Plot the bootstrap distributions of the inferred parameters of the different types as boxplots.
- Parameters:
- Returns:
Axes object
- Raises:
ValueError – If no inference objects are given or no bootstraps are found.
- plot_covariate(index: int = 0, file: str = None, show: bool = True, title: str = None, bootstrap_type: Literal['percentile', 'bca'] = 'percentile', point_estimate: Literal['original', 'mean', 'median'] = 'original', show_types: bool = True, ci_level: float = 0.05, xlabel: str = 'cov', ylabel: str = None, ax: Axes = None) Axes[source]#
Plot the covariate given by the index.
- Parameters:
bootstrap_type¶ (
Literal['percentile','bca']) – Bootstrap type.point_estimate¶ (
Literal['original','mean','median']) – Whether to use ‘original’ MLE values, ‘mean’ or ‘median’ of bootstraps as point estimate.show_types¶ (
bool) – Whether to show types on second x-axis.ci_level¶ (
float) – Tail probability per side, so 0.05 yields a 90% interval.ylabel¶ (
str) – Y-axis label, defaults to the covariate parameter name.ax¶ (
Axes) – Axes to plot on. Only for Python visualization backend.
- Returns:
Axes object.
- Raises:
ValueError – If the covariate with the given index does not exist.
- get_discretized(intervals: ndarray = array([-inf, -100., -10., -1., 0., 1., inf]), confidence_intervals: bool = True, ci_level: float = 0.05, bootstrap_type: Literal['percentile', 'bca'] = 'percentile', point_estimate: Literal['original', 'mean', 'median'] = 'mean') Tuple[Dict[str, ndarray], Dict[str, ndarray | None]][source]#
Get discretized DFEs.
- Parameters:
intervals¶ (
ndarray) – Array of interval boundaries over(-inf, inf)yieldingintervals.shape[0] - 1bins.confidence_intervals¶ (
bool) – Whether to return confidence intervalsci_level¶ (
float) – Tail probability per side, so 0.05 yields a 90% interval.bootstrap_type¶ (
Literal['percentile','bca']) – Type of bootstrappoint_estimate¶ (
Literal['original','mean','median']) – Whether to use ‘original’ MLE values, ‘mean’ or ‘median’ of bootstraps as point estimate.
- Returns:
Dictionary of values and dictionary of errors, both indexed by inference label (
marginal.<type>andjoint.<type>).
- get_stats_discretized(ci_level: float = 0.05, intervals: ndarray = array([-inf, -100., -10., -1., 0., 1., inf]), bootstrap_type: Literal['percentile', 'bca'] = 'percentile', point_estimate: Literal['original', 'mean', 'median'] = 'mean') Dict[str, Tuple[ndarray, ndarray, ndarray]][source]#
Get statistics for the discretized DFEs.
- Parameters:
ci_level¶ (
float) – Tail probability per side, so 0.05 yields a 90% interval.intervals¶ (
ndarray) – Array of interval boundaries yieldingintervals.shape[0] - 1bins.bootstrap_type¶ (
Literal['percentile','bca']) – Type of bootstrap to use.point_estimate¶ (
Literal['original','mean','median']) – Whether to use ‘original’ MLE values, ‘mean’ or ‘median’ of bootstraps as point estimate.
- Returns:
Center values, errors around center, and confidence intervals, keyed by inference label (
marginal.<type>andjoint.<type>).
- plot_bucket_sizes(*args, **kwargs)[source]#
Not available for a joint inference, whose discretization is shared by its inferences.
- Raises:
NotImplementedError – Always.
- get_bootstrap_params() Dict[str, float][source]#
Get bootstrap parameters.
- Returns:
Bootstrap parameters
- to_json() str[source]#
Serialize object. Note that the deserialized inference objects no longer share the same optimization instance among other things.
- Returns:
JSON string
- get_dfe() Dict[str, DFE][source]#
Return the inferred DFE of each type.
- Returns:
Dictionary of frozen DFE objects indexed by type.
- get_spectra() Spectra[source]#
Return the neutral, selected and modelled SFS of each type.
- Returns:
Spectra object with types
<type>.neutral,<type>.selectedand<type>.modelled.
- get_summary() Dict[str, InferenceResult][source]#
Get the summary of each type.
- Returns:
Dictionary of inference results indexed by type.
- property alpha: float | None#
There is no single alpha for the joint inference. Please refer to the
self.joint_inferences[t].alpha.- Returns:
None
- property omega: float | None#
There is no single omega for the joint inference. Please refer to the
self.joint_inferences[t].omega.- Returns:
None
- property omega_a: float | None#
There is no single omega_a for the joint inference. Please refer to the
self.joint_inferences[t].omega_a.- Returns:
None
- check_fixed_params_exist()#
Check that the fixed parameters are valid.
- compare_nested(complex: BaseInference) float | None#
Perform likelihood ratio test with given more complex model. The given model’s fixed parameters need to be a proper subset of this model’s fixed parameters. Low p-values indicate that the more complex model provides a significantly better fit.
- Parameters:
complex¶ (
BaseInference) – More complex model.- Returns:
p-value.
- Raises:
ValueError – If the inference objects are not nested.
- compare_nested_models(do_bootstrap: bool = False) Tuple[ndarray, Dict[str, BaseInference]]#
Compare the various nested versions of the specified model using likelihood ratio tests. We check for inclusion of ancestral misidentification and beneficial mutations.
- get_bootstrap_param_names() List[str]#
Get the names of the parameters to be included in the bootstraps.
- Returns:
Names of the parameters to be included in the bootstraps.
- get_bounds_linear() Dict[str, Tuple[float, float]]#
Get linear bounds for all parameters.
- Returns:
Dictionary of bounds.
- get_n_optimized() int#
Get the number of parameters that are optimized.
- Returns:
Number of parameters that are optimized.
- get_optimized_param_names() List[str]#
Get the parameters names for the parameters that are optimized.
- Returns:
List of parameter names.
- get_scales_linear() Dict[str, Literal['lin']]#
Get linear scales for all parameters.
- Returns:
Dictionary of scales.
- lrt(ll_simple: float, ll_complex: float, df: int, m: int = 0) float#
Perform a likelihood ratio test (LRT) with correction for boundary constraints as described in Molenberghs and Verbeke (2007) (https://doi.org/10.1198/000313007X171322).
- Parameters:
- Returns:
p-value
- Raises:
ValueError – If the loss function of the inference is not the likelihood.
- n_sites_div_neut: Optional[float] = None#
Number of mutational target sites over which neutral divergence was counted. Owned by the inference (not the spectra); specifying both this and
n_sites_div_selopts divergence into the likelihood.
- n_sites_div_sel: Optional[float] = None#
Number of mutational target sites over which selected divergence was counted.
- plot_interval_density(file: str = None, show: bool = True, intervals: Sequence = array([-inf, -100., -10., -1., 0., 1., inf]), interval_labels: List[str] = None, color: str = 'C0', ax: Axes = None) Axes#
Plot density of the discretization intervals chosen. Note that although this plot looks similar, this is not the DFE!
- plot_likelihoods(file: str = None, show: bool = True, title: str = 'likelihoods', scale: Literal['lin', 'log'] = 'lin', ax: Axes = None, ylabel: str = 'lnl', **kwargs) Axes#
Visualize the likelihoods of the optimization runs using a scatter plot.
- Parameters:
- Returns:
Axes object
- Raises:
ValueError – If the inference holds no likelihoods of its optimization runs.
- plot_nested_models(file: str = None, show: bool = True, remove_empty: bool = False, transpose: bool = False, cmap: str = None, title: str = 'nested model comparison', ax: Axes = None, do_bootstrap: bool = False) Axes#
Plot the p-values of nested likelihoods.
- plot_observed_sfs(labels: List[str] = None, file: str = None, show: bool = True, ax: Axes = None) Axes#
Plot neutral and selected SFS.
- runs: Optional[pd.DataFrame] = None#
Dataframe holding information on all optimization runs
- sfs_neut: Spectrum#
Neutral SFS
- sfs_sel: Spectrum#
Selected SFS
- model: Parametrization#
The DFE parametrization
- folded: bool#
Whether the SFS are folded
- n: int#
Sample size
- theta: float#
Estimate of theta from neutral SFS
- params_mle: Optional[Dict[str, float]]#
MLE estimates of the initial optimization
- sfs_mle: Optional[Spectrum]#
Modelled MLE SFS
- likelihood: Optional[float]#
Likelihood of the MLE, this value may be updated after bootstrapping
- likelihoods: Optional[List[float]]#
Likelihoods of the different ML runs, controlled by
n_runs
- n_runs: int#
Number of MLE runs to perform
- result: Optional['scipy.optimize.OptimizeResult']#
Numerical optimization result
- do_bootstrap: bool#
Whether to do bootstrapping
- n_bootstraps: int#
Number of bootstraps
- n_bootstrap_retries: int#
Number of optimization runs for each bootstrap sample
- parallelize: bool#
Whether to parallelize computations
- x0: Dict[str, Dict[str, float]]#
Initial values
- bootstraps: Optional[pd.DataFrame]#
Bootstrapped MLE parameter estimates
- bootstrap_results: Optional[List['scipy.optimize.OptimizeResult']]#
Bootstrap optimization results
- L2_residual: Optional[float]#
L2 norm of differential between modelled and observed SFS
- seed: int | None#
Random number generator seed
- rng: np.random.Generator#
Random generator instance
- execution_time: float#
Total execution time in seconds
- locked: bool#
Whether running inferences from this object is disallowed
- get_residual(k: int) float[source]#
Residual of joint inference. We calculate the residual over the jointly inferred SFS for all types.
- get_alpha(params: dict = None, use_divergence: bool = None) Dict[str, float][source]#
Get alpha, the proportion of beneficial non-synonymous substitutions, for each type.
- Parameters:
params¶ (
dict) – DFE parameters to use for calculation of format {type: {param: value}}.use_divergence¶ (
bool) – Whether to estimate alpha from observed divergence counts using polyDFE’s McDonald-Kreitman style estimator, or from the DFE alone. WhenNone(the default), this follows the inference mode. SeeBaseInference.get_alpha().
- Returns:
Dictionary of alpha indexed by type.
- get_omega(params: dict = None, use_divergence: bool = None) Dict[str, float][source]#
Get omega, the rate of non-synonymous over synonymous substitutions (dN/dS), for each type.
- Parameters:
params¶ (
dict) – DFE parameters to use for calculation of format {type: {param: value}}.use_divergence¶ (
bool) – Whether to estimate omega from observed divergence counts using polyDFE’s McDonald-Kreitman style estimator, or from the DFE alone. WhenNone(the default), this follows the inference mode. SeeBaseInference.get_omega().
- Returns:
Dictionary of omega indexed by type.
- get_omega_a(params: dict = None, use_divergence: bool = None) Dict[str, float][source]#
Get omega_a, the rate of adaptive non-synonymous over synonymous substitutions (adaptive dN/dS), for each type.
- Parameters:
params¶ (
dict) – DFE parameters to use for calculation of format {type: {param: value}}.use_divergence¶ (
bool) – Whether to estimate omega_a from observed divergence counts using polyDFE’s McDonald-Kreitman style estimator, or from the DFE alone. WhenNone(the default), this follows the inference mode. SeeBaseInference.get_omega_a().
- Returns:
Dictionary of omega_a indexed by type.
- class Inference[source]#
Bases:
objectStatic utility methods for inference objects.
Methods:
Visualize several discretized DFEs given by the list of inference objects.
Visualize several DFEs given by the list of inference objects.
Visualize the inferred parameters of several inference objects.
Visualize the bootstrap distributions of the inferred parameters of several inference objects as boxplots.
Get errors and values for MLE params of inferences.
Get values and errors of discretized DFE.
Compute errors and confidence interval for a discretized DFE.
Discretize the DFE given a DFE parametrization and its parameter values.
- static plot_discretized(inferences: List[AbstractInference], intervals: Sequence = array([-inf, -100., -10., -1., 0., 1., inf]), confidence_intervals: bool = True, ci_level: float = 0.05, bootstrap_type: Literal['percentile', 'bca'] = 'percentile', point_estimate: Literal['original', 'mean', 'median'] = 'mean', file: str = None, show: bool = True, title: str = 'discretized DFEs', labels: Sequence = None, ax: Axes = None, kwargs_legend: dict = {'prop': {'size': 8}}, **kwargs) Axes[source]#
Visualize several discretized DFEs given by the list of inference objects.
- Parameters:
inferences¶ (
List[AbstractInference]) – List of inference objects.intervals¶ (
Sequence) – Intervals over(-inf, inf)to use for discretization.confidence_intervals¶ (
bool) – Whether to plot confidence intervals.ci_level¶ (
float) – Tail probability per side, so 0.05 yields a 90% interval.bootstrap_type¶ (
Literal['percentile','bca']) – Type of bootstrap to use for confidence intervals.point_estimate¶ (
Literal['original','mean','median']) – Whether to use ‘original’ MLE values, ‘mean’ or ‘median’ of bootstraps as point estimate.kwargs¶ – Additional arguments which are ignored.
ax¶ (
Axes) – Axes to plot on. Only for Python visualization backend.kwargs_legend¶ (
dict) – Keyword arguments passed toAxes.legend(). Only for Python visualization backend.
- Returns:
Axes of the plot.
- static plot_continuous(inferences: List[AbstractInference], intervals: ndarray = array([-inf, -100., -10., -1., 0., 1., inf]), confidence_intervals: bool = True, ci_level: float = 0.05, bootstrap_type: Literal['percentile', 'bca'] = 'percentile', file: str = None, show: bool = True, title: str = 'continuous DFEs', labels: Sequence = None, scale: Literal['lin', 'log', 'symlog'] = 'lin', scale_density: bool = False, ax: Axes = None, kwargs_legend: dict = {'prop': {'size': 8}}, **kwargs) Axes[source]#
Visualize several DFEs given by the list of inference objects. By default, the PDF is plotted as is. Due to the logarithmic scale on the x-axis, we may get a wrong intuition on how the mass is distributed, however. To get a better intuition, we can optionally scale the density by the x-axis interval size using
scale_density = True. This has the disadvantage that the density now changes for x, so that even a constant density will look warped.- Parameters:
inferences¶ (
List[AbstractInference]) – List of inference objects.confidence_intervals¶ (
bool) – Whether to plot confidence intervals.ci_level¶ (
float) – Tail probability per side, so 0.05 yields a 90% interval.bootstrap_type¶ (
Literal['percentile','bca']) – Type of bootstrap to use for confidence intervals.scale¶ (
Literal['lin','log','symlog']) – y-scale of the plot.scale_density¶ (
bool) – Whether to scale the density by the x-axis interval size.ax¶ (
Axes) – Axes to plot on. Only for Python visualization backend.kwargs_legend¶ (
dict) – Keyword arguments passed toAxes.legend(). Only for Python visualization backend.kwargs¶ – Additional arguments for the plot.
- Returns:
Axes of the plot.
- static plot_inferred_parameters(inferences: List[AbstractInference], labels: Sequence, confidence_intervals: bool = True, ci_level: float = 0.05, bootstrap_type: Literal['percentile', 'bca'] = 'percentile', point_estimate: Literal['original', 'mean', 'median'] = 'mean', file: str = None, show: bool = True, title: str = 'parameter estimates', scale: Literal['lin', 'log', 'symlog'] = 'log', ax: Axes = None, kwargs_legend: dict = {'loc': 'upper right', 'prop': {'size': 8}}, **kwargs) Axes[source]#
Visualize the inferred parameters of several inference objects. Note that the DFE parametrization needs to be the same for all inference objects.
- Parameters:
inferences¶ (
List[AbstractInference]) – List of inference objects.scale¶ (
Literal['lin','log','symlog']) – y-scale of the plot.confidence_intervals¶ (
bool) – Whether to plot confidence intervals.ci_level¶ (
float) – Tail probability per side, so 0.05 yields a 90% interval.bootstrap_type¶ (
Literal['percentile','bca']) – Type of bootstrap to use for confidence intervals.point_estimate¶ (
Literal['original','mean','median']) – Whether to use ‘original’ MLE values, ‘mean’ or ‘median’ of bootstraps as point estimate.ax¶ (
Axes) – Axes to plot on. Only for Python visualization backend.kwargs_legend¶ (
dict) – Keyword arguments passed toAxes.legend(). Only for Python visualization backend.kwargs¶ – Additional arguments which are ignored.
- Returns:
Axes of the plot.
- Raises:
ValueError – If no inference objects are given.
- static plot_inferred_parameters_boxplot(inferences: List[AbstractInference], labels: Sequence, file: str = None, show: bool = True, title: str = 'parameter estimates', **kwargs) Axes[source]#
Visualize the bootstrap distributions of the inferred parameters of several inference objects as boxplots. Note that the DFE parametrization needs to be the same for all inference objects.
- Parameters:
- Returns:
Axes of the plot.
- Raises:
ValueError – If no inference objects are given or no bootstraps are found.
- static get_errors_params_mle(ci_level: float, confidence_intervals: bool, inferences: List[AbstractInference], labels: Sequence, param_names: Sequence, bootstrap_type: Literal['percentile', 'bca'], point_estimate: Literal['original', 'mean', 'median'] = 'mean') Tuple[Dict[str, Tuple[ndarray, ndarray] | None], Dict[str, ndarray]][source]#
Get errors and values for MLE params of inferences.
- Parameters:
ci_level¶ (
float) – Tail probability per side, so 0.05 yields a 90% interval.confidence_intervals¶ (
bool) – Whether to compute confidence intervals.inferences¶ (
List[AbstractInference]) – List of inference objects.param_names¶ (
Sequence) – Names of the parameters to get errors and values for.bootstrap_type¶ (
Literal['percentile','bca']) – Type of bootstrap to use.point_estimate¶ (
Literal['original','mean','median']) – Whether to use ‘original’ MLE values, ‘mean’ or ‘median’ of bootstraps as point estimate.
- Returns:
dictionary of errors and dictionary of center values indexed by labels.
- static get_discretized(inferences: List[AbstractInference], labels: Sequence, intervals: ndarray = array([-inf, -100., -10., -1., 0., 1., inf]), confidence_intervals: bool = True, ci_level: float = 0.05, bootstrap_type: Literal['percentile', 'bca'] = 'percentile', point_estimate: Literal['original', 'mean', 'median'] = 'mean') Tuple[Dict[str, ndarray], Dict[str, ndarray | None]][source]#
Get values and errors of discretized DFE.
- Parameters:
inferences¶ (
List[AbstractInference]) – List of inference objects.bootstrap_type¶ (
Literal['percentile','bca']) – Type of bootstrap to useci_level¶ (
float) – Tail probability per side, so 0.05 yields a 90% interval.confidence_intervals¶ (
bool) – Whether to compute confidence intervalsintervals¶ (
ndarray) – Array of interval boundaries over(-inf, inf)yieldingintervals.shape[0] - 1bars.point_estimate¶ (
Literal['original','mean','median']) – Whether to use ‘original’ MLE values, ‘mean’ or ‘median’ of bootstraps as point estimate.
- Returns:
Dictionary of values and dictionary of errors indexed by labels.
- static get_stats_discretized(params: dict, bootstraps: DataFrame, model: Parametrization | str, ci_level: float = 0.05, intervals: ndarray = array([-inf, -100., -10., -1., 0., 1., inf]), bootstrap_type: Literal['percentile', 'bca'] = 'percentile', point_estimate: Literal['original', 'mean', 'median'] = 'mean', warn_mass: bool = True) Tuple[ndarray, ndarray, ndarray][source]#
Compute errors and confidence interval for a discretized DFE.
- Parameters:
bootstraps¶ (
DataFrame) – Bootstrapped samplesmodel¶ (
Parametrization|str) – DFE parametrizationci_level¶ (
float) – Tail probability per side, so 0.05 yields a 90% interval.intervals¶ (
ndarray) – Array of interval boundaries yieldingintervals.shape[0] - 1bins.bootstrap_type¶ (
Literal['percentile','bca']) – Type of bootstrappoint_estimate¶ (
Literal['original','mean','median']) – Whether to use ‘original’ MLE values, ‘mean’ or ‘median’ of bootstraps as point estimate.warn_mass¶ (
bool) – Whether to warn if the CDF does not evaluate to 0 and 1 at the outer interval boundaries.
- Returns:
Center values, errors around center, and confidence intervals.
- static compute_histogram(model: Parametrization | str, params: dict, intervals: ndarray, warn_mass: bool = True) ndarray[source]#
Discretize the DFE given a DFE parametrization and its parameter values.
- Parameters:
- Returns:
Mass of the DFE in each interval, summing to 1 for intervals that cover its support
- class InferenceResult(inference: BaseInference)[source]#
Bases:
SerializableInference result.
Methods:
Load object from file.
Unserialize object.
Save object to file.
Serialize object.
Initialize object.
- __init__(inference: BaseInference)[source]#
Initialize object.
- Parameters:
inference¶ (
BaseInference) – Inference object.
-
bootstraps:
DataFrame# Bootstrap samples
-
runs:
DataFrame# Optimization runs
Bases:
objectClass specifying the sharing of params among types.
allmeans all available types or params.Example usage:
import fastdfe as fd # neutral SFS for two types sfs_neut = fd.Spectra(dict( pendula=[177130, 997, 441, 228, 156, 117, 114, 83, 105, 109, 652], pubescens=[172528, 3612, 1359, 790, 584, 427, 325, 234, 166, 76, 31] )) # selected SFS for two types sfs_sel = fd.Spectra(dict( pendula=[797939, 1329, 499, 265, 162, 104, 117, 90, 94, 119, 794], pubescens=[791106, 5326, 1741, 1005, 756, 546, 416, 294, 177, 104, 41] )) # create inference object inf = fd.JointInference( sfs_neut=sfs_neut, sfs_sel=sfs_sel, shared_params=[fd.SharedParams(types=["pendula", "pubescens"], params=["S_d"])], do_bootstrap=True ) # run inference inf.run()
Methods:
Pack shared parameters.
Unpack shared parameters.
Expand 'all' type for shared parameters.
The params to share
The types to share
Pack shared parameters. Here we extract shared parameters from type-specific keys and instead create entries with joint-types holding the shared parameters. Note that we only delete parameters from marginal types but in some cases we would have to delete them from compound types as well if a more general shared parameter were to be specified later on. We instead rely on the user not to do this.
- Parameters:
- Returns:
Packed dictionary
Unpack shared parameters. Here we extract shared parameters from joint-type keys and add them to the targeted types.
Expand ‘all’ type for shared parameters.
- class Covariate(param: str, values: Dict[str, float], callback: Callable | None = None, bounds: tuple | None = None, x0: float | None = None, bounds_scale: Literal['lin', 'log', 'symlog'] = 'symlog') None[source]#
Bases:
objectClass defining a covariate which induces a relationship with one or many parameters. The relationship is defined by a callback function which modifies the parameters. The default callback introduces a linear relationship
\[\phi_t = \phi + c (v_t - \bar{v}),\]where \(\phi_t\) is the parameter of type \(t\), \(\phi\) the parameter shared by all types, \(c\) the estimated covariate parameter, \(v_t\) the covariate value of type \(t\), and \(\bar{v}\) the mean of the covariate values over all types.
Below an example of introducing linear covariates for
S_d, the for mean strength of deleterious mutations (cf.GammaExpParametrization). Each of the three types is associated with one covariate. This we pass toJointInferencetogether with the stratified spectra:import fastdfe as fd cov = fd.Covariate( param='S_d', values=dict(type1=5, type2=3, type3=1) )
Methods:
Apply the custom or default callback to modify the given parameters.
Modify the given parameters introducing a linear relationship with the given covariate, centred at the mean covariate value.
-
bounds:
Optional[tuple] = None# The bounds of the covariate parameter to be estimated. By default, on the symmetric log scale, the upper bound is the largest absolute bound of the modified parameter divided by the largest absolute deviation of a covariate value from their mean, and the linear threshold a thousandth of it.
-
x0:
Optional[float] = None# The initial value of the covariate, by default 0 or the bound nearest to it
-
bounds_scale:
Literal['lin','log','symlog'] = 'symlog'# The scale of the bounds, on which the covariate parameter is optimized. Symlog bounds
(a, b)span the range from-btobwith linear thresholda.
- apply(covariate: float, type: str, params: Dict[str, float]) Dict[str, float][source]#
Apply the custom or default callback to modify the given parameters.
-
bounds: