Journal of Heredity 2004:95(3):217-224
© 2004 The American Genetic Association
Analytical Bayesian Approach for Assigning Individuals to Populations
From the Département Cultures Pérennes, TA 80/03, CIRAD, Avenue Agropolis, 34098 Montpellier CEDEX 5, France (Baudouin), and Centre de Biologie et de Gestion des Populations, Campus International de Baillarguet CS 30 016, 34988, Monferrier-sur-Lez CEDEX, France (Piry and Cornuet).
Address correspondence to L. Baudouin at the address above, or e-mail: Luc.baudouin{at}cirad.fr.
| Abstract |
|---|
|
|
|---|
We propose a general formulation of the Bayesian method for assigning individuals to a population among a predetermined set of reference populations using molecular marker information. Compared to previously published methods, ours allows us to consider different types of prior information about allele frequencies by using a Dirichlet prior probability distribution. It also makes it possible to assign a set of individuals assumed to belong to the same population with increased accuracy using their pooled genotype data. The efficiency of the method is illustrated by application to a group of closely related coconut populations. An interesting feature of the Bayesian procedure is the way it handles imprecise information. With a poor or even incomplete dataset, assignment is still be possible and gives valid results: poor data quality is reflected in an ambiguous result rather than in a false conclusion.
Over the last few years, various methods have been proposed for assigning individuals (or sets of individuals) to populations using molecular markers. There are various applications for such methods, for example, for assessing of genetic exchanges between populations (Banks and Eichert 2000; Paetkau et al. 1995), detection of immigrants (Rannala and Mountain 1997), managing genetic resources, or controlling poaching (Manel et al. 2002).
In the following, we will only consider methods that try to answer the question: among populations of a reference set, which is most likely to be the origin of a particular individual? (Cornuet et al. 1999). These methods are based on two assumptions that, in the considered populations, marker loci are at Hardy-Weinberg equilibrium and locus pairs are at linkage equilibrium.
Researchers who have focused on this question have had to choose between two options: in the first ("supervised" methods), the populations are given and each is represented by a certain number of individuals sampled in their environment of origin (Baudouin and Lebrun 2000). In this case, equilibrium conditions represent an approximation of reality. With "unsupervised" methods (Dawson and Belkhir 2001; Falush et al. 2003; Pritchard et al. 2000), the populations are by definition in equilibrium and characterized by the list of allele frequencies. The calculation procedure then consists of simultaneously fitting these frequencies and assigning individuals to the populations (where some individuals may descend from more than one population). This complex calculation is carried out numerically using a Monte Carlo Markov chain (MCMC) approach.
An undeniable advantage of the second option is that it makes it possible to realistically take into account complex genetic situations, including population mixes. This advantage entails two drawbacks: first, the populations are abstract objects that are not necessarily represented in the field, and this may be a hindrance for some applications. Second, numerical solutions by MCMC are demanding in computation time and require a certain degree of know-how, even when the appropriate software is available: a certain number of simulation parameters (e.g., the number of initial simulations, or "burn-in period," the number of replicates, and the number of populations) have to be determined empirically, which can be a relatively tricky operation (Chib and Greenberg 1995). Consequently, routine use of such methods is not particularly simple.
In contrast, the first option leads to calculations that remain simple enough to be treated analytically. Computations are rapid, their relevance can be checked by hand on conveniently chosen examples, and their routine use is easier. Moreover, the reference populations are effectively represented in the field and, provided populations are sufficiently isolated and the reference samples are carefully taken, the overall equilibrium conditions can be accepted, at least as an approximation. In our view, these two categories of methods should not be opposed, but seen as complementary tools: "unsupervised" MCMC methods can (should?) be used to validate the reference set of populations, whereas analytical methods are much more suited to the subsequent assignment of new individuals.
We describe here an analytical assignment procedure that generalizes previously available methods by including two extensions. One is to consider the case in which the test sample consists of several individuals. This is particularly useful in the framework of plant genetic diversity conservation: by comparing newly surveyed populations to already collected ones, gene bank managers can avoid unnecessary duplications. In that case, it is useful to consider a certain number of individuals as representative of the same unknown population. Whenever such an assumption is reasonable (which depends on reproductive behavior and on the possibility of migration), such a collective assignment procedure is expected to result in a much higher discriminating power.
The second extension is the generalization of the prior distribution of allelic frequencies (within the family of Dirichlet distributions). This makes it possible to explore the influence of the prior on the posterior probabilities and to optimize this choice. For example, the "uninformative" priors proposed so far have all their parameters
h equal, but in Rannala and Mountain (1997), the sum of the parameters is one, while Baudouin and Lebrun (2000) proposed using the uniform distribution, where all parameters are equal to one. It was thus necessary to compare these priors, referred to as R&M and UNIF, respectively, in the rest of this article. We keep in mind, however, that the choice is not limited to these priors, and "informative" priors (i.e., distributions taking available information on genetic diversity into account) will be considered in another article.
Another aim of this article is to present the problem of assignment in sufficiently general form so that it can be adapted to further applications, including the consideration of inbreeding, dominant markers such as amplified fragment length polymorphisms (AFLPs), and nondiploid genotypes.
Finally, we apply the proposed method to data observed in coconut in order to demonstrate how it can be used to reveal suspected andhighlyunsuspected relationships between populations from the Pacific coast of America.
| Materials and Methods |
|---|
|
|
|---|
Probabilistic Formulation of the Problem
The point of view adopted here is that assigning a sample (one or more individuals) to a population amounts to choosing from a set of propositions of the type "the sample belongs to population i" (proposition noted bi). This choice is made after examining the genetic structure m of the sample and those of other samples representative of a set of reference populations (which we summarize by n*). To answer, we have to express the degree of confidence we can place in each of these propositions with a probability: Pr(bi/m, n*). If this probability is nil or almost nil, we reject the proposition. However, if it is close to 1, we accept it. For an intermediate value, we conclude that the data provide ambiguous information and that we have difficulties in distinguishing between several possible origins.
In order to have a defined set of propositions from which to choose, we need to represent the diversity of the species by a series of reference populations, each of them characterized by a reference sample. We take this representation to be exhaustive and nonredundant (any individual or group of individuals sampled together belongs to one reference population and one only). This implies having good knowledge of the diversity of the species.
Principle of Bayesian Assignment
From then onward, the principle of the assignment procedure consists in applying Bayes' theorem:
|
|
|
|
|
|
The prior distribution, f(i) = Pr(bi), enables the user of a Bayesian method to take into account all the relevant data in his possession (other than the molecular data referred to in this article) regarding the identity of the tested sample. This information may arise from geographical considerations, phenotypic observations, or from the results of a previous molecular study. If this information is not available (or ignored), the statement that "all populations are equally possible" is adequately expressed by giving the same value to all Pr(bi) and equation (1) simplifies to
|
|
Likelihood Function
Let us now evaluate the probability of a sample of unknown origin belonging to a reference population. Note that when considering the variations of this probability across populations (for a given test sample), it is referred to as the likelihood of the considered population. In order to lighten the notations, we will omit hereafter the index i, which indicates the population among the set of reference populations. We therefore write Pr(m/n) for Pr(m/ni). From the definition of a conditional probability,
|
|
|
|
|
|
|
|
n) and Pr(n) in equation (3) by their values, we obtain the general result
|
|
Application to a Single Multiallelic Marker
The previous formula holds for any pair (m and n) of observations, provided they are independent from each other, but depend on the same unknown parameter vector x, characterizing the population. At present, we consider m and n as vectors whose components {mh} and {nh} represent the number of copies of allele number h in the samples at a given locus (1
h
k, where k is the number of alleles of the considered locus). We suppose that the individuals in each sample are taken at random, and that the population is in Hardy-Weinberg equilibrium. Then the probability of having m1, m2, ... , mk (respectively, n1, n2, ... , nk) copies of alleles number 1, 2, ... , k in our samples is given by the multinomial distribution
|
|
|
|
h represents the parameters of the distribution and Kx is another normalization constant (Lee 1997). Thus equation (5) becomes
|
|
|
|
|
|
are the sums of mh, nh, and
h, respectively. This probability is easily calculated if we recall that for positive real x and for a, a positive integer less than x,
(x)/
(x a) = (x 1)(x 2) ... (x a). The last step consists in applying equation (1)or equation (2) if appropriateto the above result.
Influence of the Prior
If the sample to assign consists of a single diploid individual, m is equal to 2 and equation (7) becomes Pr(m/n) = (nh +
h + 1)(nh +
h)/(n +
+ 1)(n +
) for a homozygote hh and Pr(m/n) = 2(nh +
h)(nh' +
h')/(n +
+ 1)(n +
) for a heterozygote hh'. These formulas reduce to those of Baudouin and Lebrun (2000), if all the
h are set equal to 1 and to those of Rannala and Mountain (1997) if they are set equal to 1/k. As the size of the reference sample increases, both estimators tend toward the maximum-likelihood criterion given by Paetkau et al. (1995): ph2 for homozygotes and 2.ph.ph' for heterozygotes (where ph = nh/n represents the observed frequency of allele h). However, the uncertainty of the allele frequencies in the reference samples is reflected in additional terms
h and
, respectively, in the numerator and in the denominator. As a result, the denominator remains strictly positive, even if an allele found in the tested sample is not found in the reference population and the correction of allele frequencies required in Paetkau et al. (1995) is no longer needed. Similarly the marginal probability of observing the reference sample is deduced from equation (4):
|
|
When applied to a single diploid individual (i.e., with n = 2), this expression represents a prior estimate of genotypic frequencies in a population. The UNIF prior assigns the same prior probability Pr(n) = 2/[k(k + 1)] to all genotypes, while the R&M prior assumes a higher prior probability for homozygote genotypes, Pr(n) = (k + 1)/2k2, than for heterozygotes, Pr(n) = 1/k2. If we remember that Nei's diversity index is also the expectation of the rate of heterozygosity in diploids, we see how the R&M prior differs from the UNIF prior: a lower anticipated genetic diversity will result in a lower uncertainty of posterior allele frequencies. Finally, it can be shown that the posterior probability of the vector of allele frequencies follows a Dirichlet distribution:
|
|
Application to Multilocus Genotypes
The above calculations can be generalized to several loci, provided that the populations being considered are in linkage equilibrium. In that case, the distributions of the different loci are independent and the probability of belonging to a given population is equal to the product of the probabilities computed for each locus.
More precisely, introducing another subscript for loci, the likelihood function in equations (1) and (2) becomes
|
|
Application to Coconut
As an illustration of the efficiency of the method, we applied the proposed algorithms to 74 coconuts from six populations from the Pacific coast of Latin America (Table 1). They form a group that has a narrow genetic base and is clearly distinct from the other populations of the same species.
|
The 14 microsatellite markers used were those of the microsatellite kit developed for the identification of coconut cultivars (Baudouin and Lebrun 2002). They total more than 180 alleles, 71 of which were present in the study populations. The number of alleles per locus ranged from three to eight in this study.
The algorithm described in this article was used to assign each individual to the most likely population, using the UNIF and R&M priors. An "exclusion" algorithm (Cornuet et al. 1999) based on random sample simulation was also used to detect possible outliers. The score obtained is the proportion of simulated samples (representative of the same population) whose probability, Pr(m/n), is less than or equal to that of the tested sample. These procedures were repeated after grouping the original populations according to their genetic affinities. In all those tests, the "tested" individual was excluded from the reference samples in order to avoid bias in the assignment procedure. Lastly, the benefit of grouping tested individuals was examined by splitting each group into two or three samples, one served as a reference sample, the others were tested collectively. Calculations were carried out using GeneClass2 software (Piry et al. 2004).
| Results and Discussion |
|---|
|
|
|---|
Comparison Between Prior Distributions
The algorithm provided basically similar results whichever prior was used. In the rare cases where the two priors disagreed on the most likely origin, the population preferred by one was attributed a probability of between 0.05 and 0.50 by the other. However, the UNIF prior tended to provide slightly more ambiguous (or more uncertain) results. It is worth noting that the observed rate of heterozygotes was in the same range as the one expected by the R&M prior, but notably lower than what the UNIF prior predicts, it seems that the former gave a better account of genetic diversity in the considered populations. As a result, the latter had a tendency to overestimate the uncertainty of allele frequencies (see the section on "Influence of the Prior" above). We will only comment on the results of the R&M prior.
Assigning Individuals to the Original Populations
We began by assigning all individuals to one of the first five populations. The results are summarized in Table 2, where only the population with the highest probability is taken into account. Only 53% of the corresponding probabilities were greater than 0.95 and could therefore be considered as unambiguous. Many individuals were not associated with their population of origin (23 of 70, or 33%). However, most of these cross-assignments (16 cases) occurred between the two Aguadulce populations or between Monagre and Bowden. Lastly, the four individuals from Peru were preferentially assigned to the Bowden population, but with a not insubstantial probability for Monagre.
|
The exclusion algorithm showed that the genotypes of two individuals were incompatible (p
105) with their population of origin. The first was in the Monagre population and seems to be a typical coconut from Aguadulce (both populations are planted side by side in Côte d'Ivoire). The second was in the Costa Rica population and had several alleles of unknown origin. These individuals were excluded from subsequent analyses.
Grouping Populations According to Their Genetic Affinities
Given the above results, it was clear that some of the above populations were so similar that there was no point in considering them as distinct. A first group consisted of Aguadulce populations 1 and 2. A second included the Monagre, Bowden, and Peru populations, and the Costa Rica population alone formed the third group. These groups were consistent with genetic distances (Table 3). Only three individuals were preferentially assigned to another group (Table 4). However, they were still considered as possible members of their population, according to the exclusion method, with respective scores of 0.10, 0.26, and 0.39. In addition, the assignment probabilities were greater than 0.95 for 59 individuals (81%). Finally, the cumulative distribution of the scores of the exclusion procedure was roughly linear, which suggests that the groups are sufficiently homogeneous. This improvement in the result bears witness to both the relevance of the groupings and the gain in discriminating power achieved with larger reference samples.
|
|
Grouping Tested Individuals
In order to illustrate the potential benefits of simultaneously testing several individuals from the same unknown population, we used Aguadulce 1, Monagre, and eight individuals of the Costa Rica population as reference samples, respectively, for the first, second, and third groups. When looking for the origin of the Aguadulce 2, Bowden, and Peru samples, and of the remaining seven individuals of the Costa Rica population (taken together), we obtained the correct origin with probabilities greater than 0.9999. This gain in precision can be illustrated by the Peru sample. Table 5 shows the log-likelihoods (i.e., log10(Pr(m/ni)) of the three groups for the four Peruvian individuals.
|
While the evidence in favor of group 2 was only slight when individuals were considered (log-likelihood ratio between 0.5 and 0.9, hence a likelihood ratio of less than 10), it was quite decisive (likelihood ratio of more than 104) when the four individuals were considered together. Another option was to seek the probability of simultaneously drawing these four genotypes from the reference population. This probability is proportional to the product of the individual probabilities (if the individuals are not related),
|
|
12.
Comparison with the MCMC Method
With this relatively small dataset, GeneClass2 performed the assignment procedure on our computer in less than half a second. For the "exclusion" method, testing 10,000 simulated individuals for each tested individual-reference population combination took about 10 min. This can be compared with the 15 min taken to complete 200,000 iterations of the MCMC algorithm with Structure 2.0 [see Pritchard and Wen (2002) for technical explanations about Structure 2.0]. With this software, the procedure was run several times with various burn-in periods and iteration numbers, and results were consistent except for a single run for which the logarithm of the probability of the data had an unusually large variance. Structure 2.0 identified the same three populations as GeneClass2 and the same results were obtained with and without admixture, but the effect of prior population information played a critical role: when it was used, all individuals were assigned to their own population of origin; without this information, five individuals from the Monagre population (instead of one) were assigned to the Costa Rica population. The behavior of GeneClass2 was intermediate between these extreme options because it took information on the population of origin of the reference sample for granted but ignored it for the tested sample.
| Conclusion |
|---|
|
|
|---|
The application to coconut illustrates the efficiency of the proposed procedure. In addition, it gives an example of the kind of critical examination that is required to validate a reference database for population assignment. Applying Bayesian assignment to this genetically weakly differentiated group of cross-pollinating coconut palms made it possible to precisely determine the relationships of the different samples and to group them according to their affinities. While the origin of Aguadulce 2 was confirmed, the two other populations from Panama proved difficult to distinguish, at least with the number of loci and the sample sizes used, and there was no reason to consider them as distinct. The population from Costa Rica was closely related to those from Panama, but the difference was sufficient to assign most individuals to their respective origins. Finally, the four samples from Peru were unambiguously assigned to group 2 from Panama. The assignment took on a particularly conclusive value when considering the Peru samples collectively. This close resemblance between geographically distant populations was not expected at first, and suggests that they have a common origin. Finally, the exclusion method proved useful in identifying two individuals of dubious origin.
Comparing the predicted proportion of heterozygotes to the observed one may serve as a guide to choosing a prior. The rationale is that this quantity is related to genetic diversity and thus to uncertainty in allele frequencies. In our coconut data, the R&M prior gave a better prediction of heterozygosity and was also slightly more accurate for assignment. In a future article we intend to propose more efficient priors based on the observed distribution of alleles in the species.
Interest of the Method
The method proposed above generalizes the assignment procedure proposed by Rannala and Mountain (1997) to the case where the prior distribution is a Dirichlet distribution with any parameter. Like the method proposed by Baudouin and Lebrun (2000), it can be applied when the test sample is an individual or a set of several individuals of the same origin. Grouping tested individuals considerably increased the power of the assignment test and may be recommended whenever applicable. The degree of certainty of the assignment is assessed in a natural way because the result is given in terms of probability. Compared to the unsupervised methods implemented in Structure, the proposed method required much less calculation time and was easier to use. It also concentrated on the populations as they are in the field rather than on hypothetical population structures inferred from a model including a large number of parameters. This suggests that it is well adapted to routine applications. In contrast, complex situations, implying far-from-equilibrium populations, will benefit from more elaborate techniques, using MCMC, provided the number of populations (and thus the number of parameters) is not too large.
Treatment of Missing Data
An attractive particularity of the Bayesian approach is that it simplifies the treatment of missing data. In this case, the prior distribution of allele frequencies fully plays its role. Two particular cases are worth considering: (1) A locus is missing in the test sample: all the mh are nil, as is m. In this case, the numerator and the denominator of equation (7) are equal and Pr(m/n) = 1 for all the reference populations. This reflects a self-evidence: as the locus was not observed in the test sample, it could not possibly provide information about its origin. (2) A locus is missing in reference population i. In this case, Pr(m/n) is simply equal to Pr(m) (see equation 8). Note that this probability is deduced from the prior distribution for allele frequencies only and may differ from Pr(m/n*). The nongenotyped population is thus attributed a "mean" likelihood which is consistent with the fact that in the absence of data in the reference population, nothing can be said in favor or against the sample belonging to it.
To summarize, missing data are automatically dealt with in a sensible manner by Bayesian assignment procedures. They inevitably lead to less precise assignment, but they in no way prevent calculation of the assignment scores. In contrast, methods based on maximum likelihood or genetic distances use the observed allele frequencies and the result is indeterminate as soon as data are missing in the reference population. As a result, missing data require a specific treatment.
Requirements and Limitations
It must be borne in mind that the quality of the results obtained with any assignment method depends mainly on the quality of the reference population set, which should be established using all the available information about the genetic diversity of the species. This information includes both "external" (e.g., geographical, ecological, and morphological data) and "internal" evidence [e.g., population genetics parameters, results of "unsupervised" methods and of the "exclusion method" proposed in Cornuet et al. (1999)]. The reference set should be as exhaustive as possible, each reference population should be carefully identified, in (at least approximate) equilibrium, and the number of individuals per population should be sufficient. This number varies with the precision required: for example, 10 individuals (in each reference sample as well as in the tested sample) seem to be sufficient for identifying coconut cultivars in a global context, while much larger samples will be necessary for fine-grained regional studies. Finally, the sampling procedure should be designed to avoid collecting closely related individuals.
To our knowledge, all similar methods assume that the tested individuals are not inbred: Wilson and Rannala (2003) include an inbreeding coefficient, but only at the population level. In effect, including inbreeding makes the calculations much more complicated, as Pr(m/x) is no longer in a multinomial form. Inbred (or related) individuals in a reference sample will increase the uncertainty of allele frequencies (because the inbreeding reduces the number of genes actually sampled), but will not change its expectation. If a tested individual is inbred, the likelihood of all populations will be reduced in more or less the same manner. Thus the method is likely to provide sensible results, even if the discriminating power is lower than expected. The problem is more serious with the exclusion method mentioned above, because in that case the probability of the tested sample in the population is considered for itself rather than compared to the values obtained in other populations. Thus inbreeding alone may be a cause of elimination.
Finally, the method assumes that the tested sample actually belongs to one of the reference populations and the result may be inexact if this condition is not met. It is thus recommended that the results of assignment tests be checked with the exclusion method. It must be borne in mind, however, that if the exclusion test rejects the result of assignment, several explanations are possible: the individual may come from a nonreferenced population, be a between-population hybrid, or be inbred.
The method proposed here is implemented in GeneClass2 software (Piry et al. 2004), with the two types of prior distribution. This software, which is derived from GeneClass (Cornuet et al. 1999), also proposes several other methods based on genetic distances or maximum likelihood (Paetkau et al. 1995). Each of these methods can be applied directly or through a semi-Bayesian "exclusion" procedure that compares the probability of the test individual to that of a large number of simulated samples from a population. GeneClass2 is available on the Internet at the following address: http://www.ensam.inra.fr/URLB/.
| Acknowledgments |
|---|
We are indebted to the Centre National de la Recherche Agronomique (CNRACôte d'Ivoire) and to the Coconut Industry Board (CIB Jamaica) for providing biological samples. The microsatellite kit for identification of coconut cultivars is the result of a program financed by the International Plant Genetic Resources Institute (IPGRI), the Coconut Genetic Resource Network (COGENT), and the Bureau for the Development of Research on Tropical Perennial Oil Crops (BUROTROP).
| Footnotes |
|---|
Corresponding Editor: John Burke
Received September 29, 2003
Accepted March 10, 2004
| References |
|---|
|
|
|---|
-
Banks MA, Eichert W, 2000. WHICHRUN (version 3.2): a computer program for population assignment of individuals based on multilocus genotype data. J Hered. 91:87-89.
Baudouin L, Lebrun P, 2000. An operational Bayesian approach for the identification of sexually reproduced cross-fertilized populations using molecular markers. Proceedings of the International Symposium on Molecular Markers for Characterizing Genotypes and Identifying Cultivars in Horticulture, March 69, 2000, Montpellier, France. Leuven: International Society for Horticultural Science; 8193.
Baudouin L, Lebrun P, 2002. The development of a microsatellite kit for use with coconuts. Rome: International Plant Genetic Resources Institute.
Chib S, Greenberg E, 1995. Understanding the Metropolis-Hastings algorithm. Am Stat. 49:327-335.[CrossRef]
Cornuet JM, Piry S, Luikart G, Estoup A, Solignac M, 1999. New methods employing multilocus genotypes to select or exclude populations as origins of individuals. Genetics. 153:1989-2000.
Dawson KJ, Belkhir K, 2001. A Bayesian procedure to the identification of panmictic populations and the assignment of individuals. Genet Res Camb. 78:59-77.[CrossRef][ISI][Medline]
Falush D, Stephens M, Pritchard JK, 2003. Inference of population structure using multilocus genotype data: linked loci and correlated allele frequencies. Genetics. 164:1567-1587.
Lee PM, 1997. Bayesian statistics, an introduction, 2nd ed. London: Arnold.
Manel S, Berthier P, Luikart G, 2002. Detecting wildlife poaching: identifying the origin of individuals with Bayesian assignment tests and multilocus genotypes. Conserv Biol. 16:650-659.[CrossRef]
Nei M, 1972. Genetic distance between populations. Am Nat. 106:283-291.[CrossRef][ISI]
Paetkau D, Calvert W, Stirling I, Strobeck C, 1995. Microsatellite analysis of population structure in Canadian polar bears. Mol Ecol. 4:347-354.[Medline]
Piry S, Alapetite A, Paetkau D, Cornuet J-M, Baudouin L, Estoup A, 2004. GeneClass2: a software to assign or exclude individuals to populations and detect first generation migrants. Mol Ecol Notes (in press).
Pritchard JK, Stephens M, Donnelly P, 2000. Inference of population structure using multilocus genotype data. Genetics. 155:945-959.
Pritchard JK, Wen W, 2002. Documentation for structure software, version 2. http://pritch.bsd.uchicago.edu.
Rannala B, Mountain JL, 1997. Detecting immigration by using multilocus genotypes. Proc Natl Acad Sci USA. 94:9197-9201.
Wilson GA, Rannala B, 2003. Bayesian inference of recent migration rates using multilocus genotypes. Genetics. 163:1177-1191.
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||

