Journal of Heredity Advance Access originally published online on March 10, 2005
Journal of Heredity 2005 96(4):368-375; doi:10.1093/jhered/esi039
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Estimating Heritabilities and Genetic Correlations with Marker-Based Methods: An Experimental Test in Mimulus guttatus
From the Department of Forest Sciences, University of British Columbia, 2424 Main Mall, Vancouver, British Columbia V6T 1Z4, Canada
Address correspondence to Mark van Kleunen, Institute for Biochemistry and Biology, University of Potsdam, Villa Liegnitz, Lennéstr. 7a, D-14471 Potsdam, Germany, or e-mail vkleunen{at}rz.uni-potsdam.de, kermit.ritland{at}ubc.ca.
| Abstract |
|---|
|
|
|---|
The calculation of heritabilities and genetic correlations, which are necessary for predicting evolutionary responses, requires knowledge about the relatedness between individuals. This information is often not directly available, especially not for natural populations, but can be inferred by using molecular markers such as allozymes. Several methods based on inferred relatedness from marker data have been developed to estimate heritabilities and genetic correlations in natural populations. Most methods use maximum-likelihood procedures to assign pairs or groups of individuals to predefined discrete relatedness classes (e.g., half sibs and unrelated individuals). The Ritland method, on the other hand, uses method of moments estimators to estimate pairwise relatedness among individuals as continuous values. We tested both the Ritland method and a maximum-likelihood method by applying them to a greenhouse population consisting of seed families of the herb Mimulus guttatus and comparing the results to the ones from a frequently used standard method based on half-sib families. Estimates of genetic correlations were far from accurate, especially when we used the Ritland method. However, this study shows that even with a few variable allozyme loci, it is possible to get qualitatively good indications about the presence of heritable genetic variation from marker-based methods, even though both methods underestimated it.
A trait can respond to selection only when it has heritable genetic variation and is not constrained by genetic correlations with other traits under selection (Falconer and Mackay 1996). The estimation of heritabilities and genetic correlations requires knowledge about the relatedness between individuals. This information is often not directly available, especially not for natural populations. Therefore the classical approach for estimating heritabilities and genetic correlations involves laboratory experiments with material of known pedigree, such as full-sib or half-sib families, or artificial selection experiments. However, as a consequence of differences in environmental variation between laboratory and natural conditions and of genotype-environment interactions, heritabilities and genetic correlations estimated under laboratory conditions may not be representative of the ones in the natural population of origin (e.g., Roff and Simons 1997). Therefore methods are needed to quantify the expression of genetic variation and covariation in natural, nonmanipulated populations (Mousseau 2000).
With the progress in molecular genetic techniques, new methods have been developed to study evolution in natural populations based on inferred relatedness among individuals. Most of these methods use maximum-likelihood procedures (Thompson 1975), which classify pairs of individuals into discrete relationship classes such as full sibs, half sibs, and unrelated individuals (Mousseau et al. 1998; Thomas et al. 2000) or reconstructs sibships (Thomas and Hill 2000). Several methods have been developed to calculate heritabilities based on maximum-likelihood estimates of relatedness (Mousseau et al. 1998; Thomas and Hill 2000; Thomas et al. 2000). However, these methods require prior information on the population structure, which is often not known for natural populations. This is particularly true for plant populations, where, as a consequence of open pollination, matings may occur between individuals that are related to different degrees, resulting in a population with a continuous distribution of relatedness rather than one with discrete relationship classes. Moreover, estimates based on maximum-likelihood procedures may be strongly biased when there are only few molecular markers (Thomas et al. 2000).
Ritland (1996a) used method of moments estimators for inferring the degree of relatedness among individuals along a continuum and incorporated this in a regression-based approach to estimate heritabilities and genetic correlations (Ritland 1996b). Although this approach results in larger standard errors, the estimates are generally less biased than the ones based on maximum-likelihood procedures, particularly when employing pairwise comparisons (Thomas et al. 2002). Simulation studies indicate that the accuracy of estimates from both marker-based methods can be optimized by including more samples or using a larger number of variable markers (Ritland 1996b; Thomas et al. 2000). These requirements, however, often cannot be achieved in experimental studies due to time or financial constraints.
The objective of this study was to test both the Ritland method and a maximum-likelihood method for estimating heritabilities and genetic correlations when only a relatively small number of variable marker loci are available. Therefore we grew 492 plants representing 203 seed families of Mimulus guttatus in a greenhouse. On these plants we did allozyme analysis and measured several floral traits that are related to their mating system (van Kleunen and Ritland 2004). Then we estimated heritabilities of and genetic correlations between these traits with both marker-based methods and compared them to the ones from a frequently used standard method based on estimating variance among half-sib families.
| Materials and Methods |
|---|
|
|
|---|
Study Species and Study Population
The yellow monkey flower (Mimulus guttatus; Scrophulariaceae) is an annual or perennial herb that is native to western North America and has been introduced into eastern North America, New Zealand, and Europe. The species occurs in moist habitats such as small streams, wet meadows, and on wet bluffs along the sea.
Shoots of M. guttatus consist of 0.11 m high stems that bear two opposite 15 cm long egg- or hearth-shaped leaves at each node. Side branches and single flowers are produced from meristems in the axils of the leaves. Stems may layer and root at the nodes, resulting in vegetative reproduction. The yellow, funnel-shaped, zygomorphic flowers are 14 cm in length and have conspicuous red dots at the mouth and inside of the funnel. The flowers are insect pollinated, and each fruit may produce up to 500 small seeds of about 0.02 mg.
Plant Material and Experimental Setup
The plant material used in this study was part of a larger experiment on mating system evolution in a natural population (van Kleunen and Ritland 2004). Seeds were collected from 230 plants of a large population (approximately 3000 plants) growing on a wet bluff along the sea in Lighthouse Park (49°19' N and 123°13' W), 10 km northwest of Vancouver, British Columbia.
On August 28, 2002, we sowed the seeds into 67 multitop trays, each with 36 cells (5 cm x 6 cm x 6 cm) filled with commercial potting compost, in a greenhouse with additional lighting to extend the daily light period to 16 h. For each maternal plant (seed family), we sowed 510 seeds in each of 10 randomly chosen cells with the restriction that seeds of the same seed family were not sown into more than one cell of the same tray. During the next month, we thinned seedlings to one per cell. To keep the soil permanently wet and cool, plants were automatically flooded every 2 h during the daily light period. Trays were assigned to new random positions in the greenhouse every 2 weeks until plants were too large to be moved without damaging them.
Measurements and Allozymes
From October 27 to November 1, 2002, that is, 9 weeks after sowing the seeds, we measured traits on the most recently opened flower of three randomly chosen offspring of each of the 203 seed families that had germinated. Because not all plants reached the flowering stage, the number of measured plants was 492 instead of 609. Of the 203 seed families, 30 were represented by one individual, 59 by two individuals, 112 by three individuals, and 2 by four individuals. The measured floral traits were the same ones measured in another study on mating system evolution in this species (van Kleunen and Ritland 2004): corolla width, corolla length:width ratio, anther-stigma separation, anther length, ovary length, number of red dots on the corolla, and fluctuating asymmetry (FA) in the number of red dots.
In October and November 2002, we collected fresh corollas from each measured plant for allozyme analyses using starch-gel electrophoresis, as described by van Kleunen and Ritland (2004). We scored the following eight polymorphic loci: aconotase (ACO), alcohol dehydrogenase (ADH), isocitrate dehydrogenase (IDH), malic enzyme (ME), phosphoglucoisomerase (PGI), phosphoglucomutase (PGM), and 6-phophogluconic dehydrogenase (two loci; 6PGD1, 6PGD2).
Analyses
By using the known pedigree of our greenhouse population, we could calculate heritabilities with a standard method based on partitioning of the total phenotypic variance of each trait into the variance between seed families
and the residual variance
Because the number of replicates per seed family was unbalanced, we estimated these variance components with restricted maximum-likelihood (REML) analysis of variance (ANOVA) as implemented in the statistical software GenStat (Payne et al. 1993). We estimated heritabilities as
![]() | (1) |
We also used REML analyses as implemented in the statistical software GenStat (Payne et al. 1993) to estimate genetic covariances (
SF,xy) between traits (x and y), and calculated genetic correlations between them as
![]() | (2) |
We used data of the eight allozyme loci to estimate heritabilities and genetic correlations with the Ritland method, which uses continuous values of pairwise relatedness (Ritland 1996b), and with a maximum-likelihood method, which uses discrete relatedness classes (Mousseau et al. 1998). Both methods are implemented in the publicly available computer program Mark (Ritland 2004a).
The Ritland method uses a method of moments estimator to infer the pairwise relatedness between plants from similarity in their allozyme phenotypes, as described in Ritland (1996a). Heritabilities are then estimated by regression of pairwise differences in trait values between plants on their pairwise relatedness as
![]() | (3) |
ZR is the covariance between the similarity in quantitative traits (
) and the estimated relatedness (R) between paired individuals, and
r is the actual variance of relatedness. Values of the squared pairwise relatedness outside the range 100 to 100 are excluded for the following reason: The Ritland method treats each pair of individuals as an equally informative observation, but with few gene loci, there is quite a lot of variability among pairs in their information. The first improvement one can make here is exclusion of pairs for which these estimates fall outside a range, in this case 100 to 100. That seems like an extreme range, but in actual datasets there are a few that fall way outside this range and cause enormous distortion of the population estimate.
We also used the Mark computer program to estimate genetic correlations as
![]() | (4) |
ov(C, R) is the covariance between the pairwise relatedness (R) and the phenotypic covariances (C) between individuals (i and j) for each trait (Cxixj and Cyiyj) and a combination of traits (Cxiyj) calculated from all possible pairs of individuals. The maximum-likelihood method first uses the allozyme data to classify pairs of individuals in predefined discrete relationship classes, and then combines this information with the quantitative trait data in a mixture model to infer heritabilities and genetic correlations (Mousseau et al. 1998). In accordance with the assumptions used in the analyses with the standard method, the relationship classes in our analyses corresponded to pairs of half sibs and pairs of unrelated individuals.
Significance levels and standard errors of heritabilities and genetic correlations from both marker-based methods were determined from distributions of 1000 bootstrap values, in which individuals were the unit of bootstrapping. To test for similarity between the heritabilities estimated with both marker-based methods and the standard method, we calculated Pearson's correlations between them. We tested the similarity between the matrices of genetic correlations from the three methods with Mantel tests by using the Mantel software (Liedloff 1999).
| Results |
|---|
|
|
|---|
Allozyme Variation
Of the eight polymorphic allozyme loci, half were rather variable (allele frequencies, ACO: 0.776, 0.224; ADH: 0.503, 0.497; ME: 0.720, 0.265, 0.016; 6PGD2: 0.575, 0.425), while the other half were rather invariable (IDH: 0.961, 0.039; 6PGD1: 0.954, 0.046; PGI: 0.948, 0.052; PGM: 0.905, 0.095).
Inferred Relatedness
In the analyses using the Ritland method, the mean ± standard error (SE) coefficient of kinship, which equals half the coefficient of relatedness (Ritland 1996a), between pairs of presumed half sibs was 0.119 ± 0.013, and was not statistically different from the expected value of 0.125 (t361 = 0.423, P = .668). Moreover, it was significantly higher than the coefficient of kinship between pairs of presumed unrelated individuals, which was 0.002 ± 0.001 (t361 = 9.373, P < .001).
The variance in relatedness over all pairwise combinations of presumed half sibs (362 pairs with a coefficient of kinship of 0.125) and unrelated individuals (120,424 pairs with a coefficient of kinship of 0) was only 0.00005. However, the variance of actual relatedness between individuals as estimated from the allozyme data with the Ritland method was 0.009 and significant (P < .001).
In the estimates from the maximum-likelihood method, 2.3% of all possible pairs of individuals were assigned as half sibs and the remaining as being unrelated. This exceeds the percentage of presumed half sibs of 0.3% (P < .001 as determined from bootstrap distribution).
Heritability Estimates
Heritabilities estimated with the standard method were significant for all floral traits with the exception of fluctuating asymmetry in the number of red dots per flower (Table 1). Heritabilities estimated with the Ritland method were about four times smaller than the ones estimated with the standard method (Figure 1, Table 1). Nevertheless, most of the Ritland estimates were significant and positively correlated with the ones from the standard method (Figure 1; r = 0.752, one-sided P = .026, N = 7). Heritabilities estimated with the maximum-likelihood method also underestimated the heritabilities of the standard method, but tended to be higher than the ones estimated with the Ritland method, especially for traits that also had high Ritland estimates (Table 1). Heritabilities estimated with the maximum-likelihood method were also positively correlated with the ones from the standard method (Figure 1; r = 0.679, one-sided P = .047, N = 7).
|
|
Estimates of Genetic Correlation
Eight of the 21 genetic correlations between floral traits estimated with the standard method based on half-sib families were significant or marginally significant (Table 2, panel A). Three of the genetic correlations were also significant when estimated with the Ritland method. However, all genetic correlations estimated with the Ritland method, including the significant ones, had large standard errors that greatly exceeded the actual estimates (Table 2, panel B). Moreover, three of the nonsignificant ones lay far outside of the biologically possible range of 1 to 1 (Table 2, panel B). As a consequence of the inaccuracy of these estimates, there was no similarity between the matrix of genetic correlations estimated with the standard method and the one estimated with the Ritland method (Figure 2; r = 0.354, Mantel Z = 2.755, one-sided P = .962).
|
|
On the other hand, 9 of the 21 genetic correlations estimated with the maximum-likelihood method were significant or marginally significant, and had much smaller standard errors than the ones from the Ritland method (Table 2, panel B). Six of the significant genetic correlations were also significant when estimated with the standard method (Table 2, panel A). Moreover, even though the genetic correlations estimated with the maximum-likelihood method were smaller in magnitude than the ones estimated with the standard method, they were positively correlated (Figure 2; r = 0.734, Mantel Z = 1.851, one-sided P = .002).
| Discussion |
|---|
|
|
|---|
Estimates of Heritability
One of the main constraints in using the Ritland method for estimating heritabilities is that it requires variation in the actual degree of relatedness (Ritland 1996b). Often this may be hard to detect with molecular markers in natural populations (Ritland and Ritland 1996; van Kleunen and Ritland 2004). Although in this study the variation in the actual degree of relatedness was relatively small, it was significant and sufficiently large to estimate heritabilities.
The relatedness inferred from allozyme data with the Ritland method for pairs of presumed half sibs and unrelated individuals were close to the expected values, indicating that estimates of pairwise relatedness with the Ritland method are quite accurate. The variance in the actual degree of relatedness, however, exceeded the one based on presumed relatedness because there was variation in inferred relatedness among pairs of presumed half sibs and among pairs of presumed unrelated individuals. This might partly reflect estimation error, as indicated by negative values for some of the estimates. On the other hand, it may indicate true variation in relatedness among presumed half-sibs and among presumed unrelated individuals. Half sibs may have received, by chance, different sets of alleles from their mother. Further, some of the presumed half sibs may have a higher than expected relatedness because they actually share the same father (i.e., are full sibs) or because their parents are related. The latter may also be true for some of the pairs of presumed unrelated individuals. Moreover, although they do not share the same mother, they may still share the same father (i.e., paternal half sibs). These arguments may also explain why the proportion of half sibs as estimated with the maximum-likelihood method exceeded the presumed proportion of half sibs by a factor of 10.
Both the Ritland and maximum-likelihood method gave heritability estimates that were smaller than the ones calculated with the standard method based on half-sib seed families. Simulations also showed that heritabilities estimated with the maximum-likelihood method that we used (i.e., the one developed by Mousseau et al. [1998]) are generally biased downward (Thomas et al. 2000), and that the ones from the Ritland method are usually biased upward (Ritland 1996b; Thomas et al. 2002), but downward when the actual variation in relatedness is low, as was the case in our study (Thomas et al. 2000). In another experimental study, heritabilities estimated with the Ritland method in a natural population of M. guttatus were also higher than the ones estimated from regression of greenhouse-grown offspring on field-grown parents (Ritland and Ritland 1996).
Two of the seven heritability estimates from the standard method exceeded the biologically possible maximum value of one (Table 1). This suggests that part of the deviation of the marker-based estimates from those of the standard method in our study might be a consequence of overestimation of the true heritability by the standard method. The variation among maternal half-sib families may have been inflated by maternal carryover effects, and as a consequence the genetic variance may have been overestimated with the standard method (Falconer and Mackay 1996). This means that the discrepancy between the heritability estimates from the marker-based methods and the true heritability might be smaller than the estimates from the standard method suggest.
Although the heritability estimates from the marker-based methods were lower in magnitude than those from the standard method, they were positively correlated with each other (Figure 1). On the other hand, in the study of Ritland and Ritland (1996), there was no significant correlation between heritabilities estimated with the Ritland method and those estimated from offspring-parent regression (r = 0.181, P = .535, N = 14). This might be due to the fact that the greenhouse environment of the offspring and the natural environment of the parents were different, or due to a sampling effect because only half of the plants used in the Ritland method were represented in the offspring-parent regression. Another explanation could be that heritabilities calculated with the Ritland method are not reliable. Our study shows, however, that even though heritabilities estimated with the Ritland method underestimate the true magnitude of the heritability, they give a qualitatively good impression of whether there is heritable genetic variation in a trait.
There certainly remains a need for more accurate methods to estimate heritabilities and genetic correlations in the absence of pedigree information. However, even the possibility of determining the presence of heritable genetic variation is a big improvement for studies testing for the potential for evolution in natural populations. On the other hand, in natural populations, results from marker-based methods should be interpreted with even more caution than in our controlled experiment. When, in natural populations, closely related individuals tend to grow closer together than unrelated individuals, phenotypic resemblance may not only be caused by relatedness, but also by shared environments. This might bias estimates of heritability with a marker-based method, as applied in our study. However, methods have been developed to correct for this potential bias by using models that either allow for a constant amount of shared environment or for a linear decline in shared environment with distance (Ritland 1996b).
Estimates of Genetic Correlation
Genetic correlations are notoriously difficult to estimate accurately because they require accurate estimates of the genetic variances of the traits and the genetic covariances between them (Lynch 1999). Genetic correlations estimated with the standard method based on a large number of half-sib families had high standard errors, and only 8 of 21 were significant (Table 2). Genetic correlations estimated with the Ritland method had even larger standard errors, and several estimates were outside of the biologically possible range of 1 to 1, which indicates that the estimates were unstable. The reasons for this are not clear, but may be a consequence of the low number of variable loci used in this study. Moreover, the large standard errors of the estimates may partly be a consequence of bootstrapping individuals instead of pairs of individuals. While the latter would underestimate the standard error, the first may overestimate it by 10100% (Thomas et al. 2002). This, however, should also be true for the heritability estimates that did not have such large standard errors. Nevertheless, the three genetic correlations estimated with the Ritland method that were significant were also significant when estimated with the standard method. This suggests that when significant genetic correlations are found with the Ritland method, at least the sign of the correlation and its significance are reliable.
The maximum-likelihood method performed better than the Ritland method in estimating genetic correlations. All its estimates lay within the biologically possible range and had relatively small standard errors. Moreover, these estimates were positively correlated with the ones from the standard method. However, the genetic correlations estimated with the maximum-likelihood method were smaller than those estimated with the standard method. This indicates that the estimates indicate the presence of genetic correlations, not their magnitude.
| Conclusion |
|---|
|
|
|---|
This study shows that the estimation of heritabilities and genetic correlations using molecular markers is still problematic and requires further development of more accurate methods. Although the Ritland and maximum-likelihood methods gave similar results for the heritability estimates, the maximum-likelihood method appeared to perform better, though still far from optimal, in estimating genetic correlations. This result, and the results of other studies (Thomas et al. 2000, 2002), suggests maximum-likelihood methods rather than the Ritland method should be used to estimate quantitative genetic parameters in natural populations. However, maximum-likelihood methods can only be applied when there is prior knowledge of the population structure, as was the case in our study, as well as another experimental study on sheep (Thomas et al. 2002) and a simulation study (Thomas et al. 2000) comparing marker-based methods. In natural populations, however, the population structure is often not known and relatedness is more likely to span a continuum rather than consist of discrete relatedness classes. Therefore the Ritland method seems to be the best choice for inferring the presence of heritable genetic variation in natural populations, even though the estimates are not accurate, at least not when only a few variable marker loci are available.
| Acknowledgments |
|---|
We thank T. Wahbe for practical assistance, C. Ritland and H. Wellham for teaching isozyme methods, and J. Willis for helpful comments on an earlier version of the manuscript. This research was supported by a postdoctoral fellowship award (to M.K.) by the Swiss National Science Foundation and by a Natural Sciences and Engineering Research Council of Canada discovery grant (to K.R.).
| Footnotes |
|---|
Corresponding Editor: John M. Burke
Received March 28, 2004
Accepted November 5, 2004
| References |
|---|
|
|
|---|
-
Falconer DS Mackay TFC, 1996. Introduction to quantitative genetics, 4th ed Essex: Longman.
Liedloff A, 1999. Mantel nonparametric test calculator, version 2.00 Brisbane: Queensland University of Technology.
Littell RC, Milliken GA, Stroup WW, and Wolfinger RD, 1996. SAS system for mixed models Cary, NC: SAS Institute.
Lynch M, 1999. Estimating genetic correlations in natural populations. Genet Res 74:255264.[CrossRef][Web of Science][Medline]
Morrell CH, 1998. Likelihood ratio testing of variance components in the linear mixed effects model using restricted maximum likelihood. Biometrics 54:15601568.[Medline]
Mousseau TA, 2000. Intra- and interpopulation genetic variation In: Adaptive genetic variation in the wild (Mousseau TA, Sinervo B, and Endler J, eds). New York: Oxford University Press; 219250.
Mousseau TA, Ritland K, and Heath DD, 1998. An novel method for estimating heritability using molecular markers. Heredity 80:218244.[CrossRef]
Payne RW, Lane PW, Digby PGN, Harding SA, Leech PK, Morgan GW, Todd AD, Thompson R, Tunnicliffe Wilson G, Welham SJ et al., 1993. GenStat 5, release 3, reference manual Oxford: Clarendon Press.
Ritland K, 1989. Correlated matings in the partial selfer Mimulus guttatus. Evolution 43:848859.[CrossRef]
Ritland K, 1996a. Estimators for pairwise relatedness and individual inbreeding coefficients. Genet Res 67:175185.[Web of Science]
Ritland K, 1996b. A marker-based method for inferences about quantitative inheritance in natural populations. Evolution 50:10621073.[CrossRef]
Ritland K, 2002. Extensions of models for the estimation of mating systems using n independent loci. Heredity 88:221228.[CrossRef][Web of Science][Medline]
Ritland K, 2004a. Mark, version 2.0 (last modified August 2004) Vancouver, British Columbia: Department of Forest Sciences. Available at http://www.genetics.forestry.ubc.ca/ritland/programs.html.
Ritland K, 2004b. MLTR for Windows, version 3.0 (last modified September 2004) Vancouver, British Columbia: Department of Forest Sciences. Available at http://www.genetics.forestry.ubc.ca/ritland/programs.html.
Ritland K and Ritland C, 1996. Inferences about quantitative inheritance based on natural population structure in the yellow monkeyflower, Mimulus guttatus. Evolution 50:10741082.[CrossRef]
Roff DA and Simons AM, 1997. The quantitative genetics of wing dimorphism under laboratory and "field" conditions in the cricket Gryllus pennsylvanicus. Heredity 78:235240.[CrossRef]
Thomas SC, Coltman DW, and Pemberton JM, 2002. The use of marker-based relationship information to estimate the heritability of body weight in a natural population: a cautionary tale. J Evol Biol 15:9299.[CrossRef]
Thomas SC and Hill WG, 2000. Estimating quantitative genetic parameters using sibships reconstructed from marker data. Genetics 155:19611972.
Thomas SC, Pemberton JM, and Hill WG, 2000. Estimating variance components in natural populations using inferred relationships. Heredity 84:427436.
Thompson EA, 1975. The estimation of pairwise relationships. Ann Hum Genet 39:173188.[Web of Science][Medline]
van Kleunen M and Ritland K, 2004. Predicting evolution of floral traits associated with mating system in a natural plant population. J Evol Biol 17:13891399.[Medline]
This article has been cited by other articles:
![]() |
F. D Frentiu, S. M Clegg, J. Chittock, T. Burke, M. W Blows, and I. P.F Owens Pedigree-free animal models: the relatedness matrix reloaded Proc R Soc B, March 22, 2008; 275(1635): 639 - 647. [Abstract] [Full Text] [PDF] |
||||
![]() |
T. Shikano Estimation of Quantitative Genetic Parameters Using Marker-Inferred Relatedness in Japanese Flounder: A Case Study of Upward Bias J. Hered., March 1, 2008; 99(2): 94 - 104. [Abstract] [Full Text] [PDF] |
||||
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||







