您现在访问的是微软AZURE全球版技术文档网站,若需要访问由世纪互联运营的MICROSOFT AZURE中国区技术文档网站,请访问 https://docs.azure.cn.

水平数据分区、垂直数据分区和功能数据分区Horizontal, vertical, and functional data partitioning

在许多大型解决方案中,数据划分为多个 分区 ,可以单独进行管理和访问。In many large-scale solutions, data is divided into partitions that can be managed and accessed separately. 分区可以改善可伸缩性、减少争用,以及优化性能。Partitioning can improve scalability, reduce contention, and optimize performance. 另外,它还能提供一种按使用模式来分割数据的机制。It can also provide a mechanism for dividing data by usage pattern. 例如,可以将较旧的数据存档在成本较低的数据存储中。For example, you can archive older data in cheaper data storage.

但是,必须慎重选择分区策略,才能最大程度地提高效益,将负面影响降到最低。However, the partitioning strategy must be chosen carefully to maximize the benefits while minimizing adverse effects.


在本文中,术语“分区”是指以物理方式将数据分割成独立数据存储的过程。In this article, the term partitioning means the process of physically dividing data into separate data stores. 它与 SQL Server 表分区不同。It is not the same as SQL Server table partitioning.

为何要将数据分区?Why partition data?

  • 提高缩放性Improve scalability. 纵向扩展单一数据库系统最终会达到物理硬件的限制。When you scale up a single database system, it will eventually reach a physical hardware limit. 如果跨多个分区来分割数据,则每个分区托管在独立的服务器上,使系统几乎能够无限横向扩展。If you divide data across multiple partitions, each hosted on a separate server, you can scale out the system almost indefinitely.

  • 改善性能Improve performance. 在每个分区上的数据访问操作通过较小的数据卷进行。Data access operations on each partition take place over a smaller volume of data. 在正确操作的情况下,分区可以提高系统的效率。Correctly done, partitioning can make your system more efficient. 影响多个分区的操作可以同时运行。Operations that affect more than one partition can run in parallel.

  • 提高安全性Improve security. 在某些情况下,可以将敏感数据和 nonsensitive 数据分隔到不同的分区,并将不同的安全控件应用于敏感数据。In some cases, you can separate sensitive and nonsensitive data into different partitions and apply different security controls to the sensitive data.

  • 提供操作灵活性Provide operational flexibility. 分区为微调操作提供了许多机会,最大限度地提高了管理效率,并最大限度降低了成本。Partitioning offers many opportunities for fine-tuning operations, maximizing administrative efficiency, and minimizing cost. 例如,可以根据数据在每个分区中的重要性定义不同的策略,以管理、监视、备份和还原及其他管理任务。For example, you can define different strategies for management, monitoring, backup and restore, and other administrative tasks based on the importance of the data in each partition.

  • 将数据存储和使用模式相匹配Match the data store to the pattern of use. 分区允许根据数据存储提供的成本和内置功能,将每个分区部署在不同类型的数据存储上。Partitioning allows each partition to be deployed on a different type of data store, based on cost and the built-in features that data store offers. 例如,大型二进制数据可存储在 Blob 存储中,而结构化程度更高的数据则可保存在文档数据库中。For example, large binary data can be stored in blob storage, while more structured data can be held in a document database. 请参阅选择适当的数据存储See Choose the right data store.

  • 提高可用性Improve availability. 跨多个服务器隔离数据可避免单点故障。Separating data across multiple servers avoids a single point of failure. 如果一个实例发生故障,只有该分区中的数据不可用。If one instance fails, only the data in that partition is unavailable. 其他分区上的操作可以继续进行。Operations on other partitions can continue. 对于托管型 PaaS 数据存储,则不怎么需要进行这种考虑,因为这些服务在设计时已经考虑到了内置冗余。For managed PaaS data stores, this consideration is less relevant, because these services are designed with built-in redundancy.

设计分区Designing partitions

数据分区有三个典型策略:There are three typical strategies for partitioning data:

  • 水平分区\(通常称为 分片)。Horizontal partitioning (often called sharding). 在此策略中,每个分区都是独立的数据存储,但所有分区具有相同的架构。In this strategy, each partition is a separate data store, but all partitions have the same schema. 每个分区称为分片,保存数据的特定子集,例如特定的一组客户的所有订单。Each partition is known as a shard and holds a specific subset of the data, such as all the orders for a specific set of customers.

  • 垂直分区Vertical partitioning. 在此策略中,每个分区在数据存储中保存项字段的子集。In this strategy, each partition holds a subset of the fields for items in the data store. 这些字段已根据其使用模式进行分割。The fields are divided according to their pattern of use. 例如,将经常访问的字段放在一个垂直分区,将较不经常访问的字段放在另一个垂直分区。For example, frequently accessed fields might be placed in one vertical partition and less frequently accessed fields in another.

  • 功能分区Functional partitioning. 在此策略中,数据已根据系统中每个界限上下文使用数据的方式进行聚合。In this strategy, data is aggregated according to how it is used by each bounded context in the system. 例如,电子商务系统可能在一个分区中存储发票数据,在另一个分区中存储产品库存数据。For example, an e-commerce system might store invoice data in one partition and product inventory data in another.

这些策略可以组合起来使用,建议在设计分区方案时全盘考虑。These strategies can be combined, and we recommend that you consider them all when you design a partitioning scheme. 例如,可以将数据分割成分片,然后使用垂直分区进一步细分每个分片中的数据。For example, you might divide data into shards and then use vertical partitioning to further subdivide the data in each shard.

水平分区(分片)Horizontal partitioning (sharding)

图 1 显示了水平分区或分片。Figure 1 shows horizontal partitioning or sharding. 在此示例中,产品库存数据已根据产品键分割成分片。In this example, product inventory data is divided into shards based on the product key. 每个分片保存分区键(A-G 和 H-Z)的连续范围数据,根据字母顺序排列。Each shard holds the data for a contiguous range of shard keys (A-G and H-Z), organized alphabetically. 分片可以将负载分散到多台计算机,减少争用并改善性能。Sharding spreads the load over more computers, which reduces contention and improves performance.


图 1-基于分区键对 (分片) 数据进行水平分区。Figure 1 - Horizontally partitioning (sharding) data based on a partition key.

最重要的因素是分片键的选择。The most important factor is the choice of a sharding key. 系统运行之后,就很难更改该键。It can be difficult to change the key after the system is in operation. 键必须确保在进行数据分区时,工作负荷能够尽量跨分片平均分配。The key must ensure that data is partitioned to spread the workload as evenly as possible across the shards.

分片不需大小一致。The shards don't have to be the same size. 请求数的平衡更为重要。It's more important to balance the number of requests. 有些分片可能非常大,但每个项的访问操作数目少。Some shards might be very large, but each item has a low number of access operations. 其他分片可能较小,但是更常访问每个项。Other shards might be smaller, but each item is accessed much more frequently. 另一个重点是确保单个分片不超过数据存储的规模限制(在容量和资源处理方面)。It's also important to ensure that a single shard does not exceed the scale limits (in terms of capacity and processing resources) of the data store.

避免产生“热”分区,否则可能会影响性能与可用性。Avoid creating "hot" partitions that can affect performance and availability. 例如,使用客户名称的第一个字母会导致不均衡分布,因为某些字母更常见。For example, using the first letter of a customer's name causes an unbalanced distribution, because some letters are more common. 相反,请使用客户标识符的哈希跨分区更均匀地分布数据。Instead, use a hash of a customer identifier to distribute data more evenly across partitions.

选择分片键时,应尽量避免将来还需要把大分片拆分、把小分片合并成大分区或者更改架构的情况。Choose a sharding key that minimizes any future requirements to split large shards, coalesce small shards into larger partitions, or change the schema. 这些操作可能非常耗时,并且可能需要在执行时使一个或多个分片脱机。These operations can be very time consuming, and might require taking one or more shards offline while they are performed.

如果复制分片,则某些副本也许能够保持联机,而其他副本会被拆分、合并,或者重新配置。If shards are replicated, it might be possible to keep some of the replicas online while others are split, merged, or reconfigured. 但是,系统可能需要限制可以在重新配置期间执行的操作。However, the system might need to limit the operations that can be performed during the reconfiguration. 例如,可以将副本中的数据标记为只读,避免出现数据不一致的情况。For example, the data in the replicas might be marked as read-only to prevent data inconsistences.

有关水平分区的详细信息,请参阅 分片模式For more information about horizontal partitioning, see sharding pattern.

垂直分区Vertical partitioning

垂直分区的最常见用途是降低与提取频繁访问的项相关的 I/O 和性能成本。The most common use for vertical partitioning is to reduce the I/O and performance costs associated with fetching items that are frequently accessed. 图 2 显示了垂直分区的示例。Figure 2 shows an example of vertical partitioning. 在此示例中,项的不同属性存储在不同的分区中。In this example, different properties of an item are stored in different partitions. 一个分区保存经常访问的数据,包括产品名称、说明和价格。One partition holds data that is accessed more frequently, including product name, description, and price. 另一个分区保存清单数据:库存计数和上次订购日期。Another partition holds inventory data: the stock count and last-ordered date.


图 2-按使用模式将数据垂直分区。Figure 2 - Vertically partitioning data by its pattern of use.

在此示例中,应用程序在向客户显示产品详细信息时,按常规查询产品名称、描述和价格。In this example, the application regularly queries the product name, description, and price when displaying the product details to customers. 库存计数和上次订购日期保存在单独的分区中,因为这两项通常一起使用。Stock count and last- ordered date are held in a separate partition because these two items are commonly used together.

垂直分区的其他优势:Other advantages of vertical partitioning:

  • 不怎么变化的数据(产品名称、说明和价格)和较动态的数据(库存水平和上次订购日期)可以彼此隔离。Relatively slow-moving data (product name, description, and price) can be separated from the more dynamic data (stock level and last ordered date). 应用程序可以将不怎么变化的数据缓存在内存中。Slow moving data is a good candidate for an application to cache in memory.

  • 可以将敏感数据存储在单独的分区中并施加额外的安全控制。Sensitive data can be stored in a separate partition with additional security controls.

  • 垂直分区可以减少必需的并发访问数量。Vertical partitioning can reduce the amount of concurrent access that's needed.

垂直分区在数据存储中的实体级运行,会部分规范化某个实体,以将它从 项分割成一组 项。Vertical partitioning operates at the entity level within a data store, partially normalizing an entity to break it down from a wide item to a set of narrow items. 在理想的情况下,垂直分区适用于 HBase 和 Cassandra 等列导向型数据存储。It is ideally suited for column-oriented data stores such as HBase and Cassandra. 如果列集合中的数据不太可能会更改,还可以考虑使用 SQL Server 中的列存储。If the data in a collection of columns is unlikely to change, you can also consider using column stores in SQL Server.

功能分区Functional partitioning

如果可以在应用程序中确定每个不同商业领域的受限上下文,则可通过功能分区来改善隔离效果和数据访问性能。When it's possible to identify a bounded context for each distinct business area in an application, functional partitioning is a way to improve isolation and data access performance. 功能分区的另一种常见用途是将读写数据与只读数据隔离。Another common use for functional partitioning is to separate read-write data from read-only data. 图 3 显示了功能分区的概览,其中的库存数据已与客户数据相隔离。Figure 3 shows an overview of functional partitioning where inventory data is separated from customer data.


图 3-按界限上下文或子域对数据进行功能分区。Figure 3 - Functionally partitioning data by bounded context or subdomain.

此分区策略可帮助减少跨系统中不同部件的数据访问争用。This partitioning strategy can help reduce data access contention across different parts of a system.

针对缩放性设计分区Designing partitions for scalability

请务必考虑每个分区的大小和工作负荷并进行平衡,使数据分布实现最大伸缩性。It's vital to consider size and workload for each partition and balance them so that data is distributed to achieve maximum scalability. 但是,还必须将数据分区,使它不超过单个分区存储的缩放限制。However, you must also partition the data so that it does not exceed the scaling limits of a single partition store.

在针对缩放性设计分区时,请执行以下步骤:Follow these steps when designing partitions for scalability:

  1. 分析应用程序以了解数据访问模式,例如每个查询返回的结果集大小、访问的频率、固有的延迟,以及服务器端计算处理要求。Analyze the application to understand the data access patterns, such as the size of the result set returned by each query, the frequency of access, the inherent latency, and the server-side compute processing requirements. 在许多情况下,一些主要实体需要大部分的处理资源。In many cases, a few major entities will demand most of the processing resources.
  2. 使用这种分析来确定当前和将来的缩放性目标,例如数据大小和工作负荷。Use this analysis to determine the current and future scalability targets, such as data size and workload. 然后将数据跨分区分布以符合缩放性目标。Then distribute the data across the partitions to meet the scalability target. 对于水平分区,选择适当的分片键很重要,可以确保数据的均衡分配。For horizontal partitioning, choosing the right shard key is important to make sure distribution is even. 有关详细信息,请参阅 分片模式For more information, see the sharding pattern.
  3. 确保每个分区有足够的资源,能够在数据大小和吞吐量方面应对可伸缩性要求。Make sure each partition has enough resources to handle the scalability requirements, in terms of data size and throughput. 每个分区的存储空间量、处理能力或网络带宽可能存在限制,具体取决于数据存储的情况。Depending on the data store, there might be a limit on the amount of storage space, processing power, or network bandwidth per partition. 如果可伸缩性要求会超出这些限制,则可能需要优化分区策略或进一步拆分数据,也许需要将至少两个策略组合在一起。If the requirements are likely to exceed these limits, you may need to refine your partitioning strategy or split data out further, possibly combining two or more strategies.
  4. 对系统进行监视,以验证数据是否按预期分配,以及分区是否可以处理负载。Monitor the system to verify that data is distributed as expected and that the partitions can handle the load. 实际使用情况并非始终符合分析的预期。Actual usage does not always match what an analysis predicts. 如果出现这种情况,可以对分区进行再平衡,或者重新设计系统的某些部分,以实现所需的平衡。If so, it might be possible to rebalance the partitions, or else redesign some parts of the system to gain the required balance.

某些云环境会根据基础结构边界分配资源。Some cloud environments allocate resources in terms of infrastructure boundaries. 请确保所选边界的限制可在数据存储、处理能力和带宽等方面提供足够的空间,以满足数据量的预期增长。Ensure that the limits of your selected boundary provide enough room for any anticipated growth in the volume of data, in terms of data storage, processing power, and bandwidth.

例如,如果使用 Azure 表存储,则单一分区在特定时限内能够处理的请求数会存在限制。For example, if you use Azure table storage, there is a limit to the volume of requests that can be handled by a single partition in a particular period of time. (有关详细信息,请参阅 Azure 存储可伸缩性和性能目标。 ) 繁忙的分片可能需要比单个分区可处理的资源更多的资源。(For more information, see Azure storage scalability and performance targets.) A busy shard might require more resources than a single partition can handle. 如果属于这种情况,可能需要对分片进行重新分区以分散负载。If so, the shard might need to be repartitioned to spread the load. 如果这些表的总大小或吞吐量超过存储帐户的容量,可能需要创建其他存储帐户并跨帐户分散表。If the total size or throughput of these tables exceeds the capacity of a storage account, you might need to create additional storage accounts and spread the tables across these accounts.

针对查询性能设计分区Designing partitions for query performance

使用较小的数据集和运行并行查询通常可以提高查询性能。Query performance can often be boosted by using smaller data sets and by running parallel queries. 每个分区应该包含整个数据集的一小部分。Each partition should contain a small proportion of the entire data set. 这种数量缩减可以提高查询性能。This reduction in volume can improve the performance of queries. 但是,分区并不是合理设计和配置数据库的替代方式。However, partitioning is not an alternative for designing and configuring a database appropriately. 例如,请确保编制必要的索引。For example, make sure that you have the necessary indexes in place.

在针对查询性能设计分区时,请执行以下步骤:Follow these steps when designing partitions for query performance:

  1. 检查应用程序的要求和性能:Examine the application requirements and performance:

    • 根据业务要求来确定必须始终快速执行的重要查询。Use business requirements to determine the critical queries that must always perform quickly.
    • 监视系统以识别任何执行速度缓慢的查询。Monitor the system to identify any queries that perform slowly.
    • 找出最常执行的查询。Find which queries are performed most frequently. 即使单个查询的开销极少,也会出现资源累积消耗很大的情况。Even if a single query has a minimal cost, the cumulative resource consumption could be significant.
  2. 将导致性能变慢的数据分区。Partition the data that is causing slow performance:

    • 限制每个分区的大小,使查询响应时间在目标范围内。Limit the size of each partition so that the query response time is within target.
    • 如果使用水平分区,请在设计分片键时,确保应用程序能够轻松地选择适当的分区。If you use horizontal partitioning, design the shard key so that the application can easily select the right partition. 这可防止查询需要扫描每个分区。This prevents the query from having to scan through every partition.
    • 考虑分区的位置。Consider the location of a partition. 如果可能,请尽量将数据保留在地理位置靠近访问数据的应用程序和用户的分区中。If possible, try to keep data in partitions that are geographically close to the applications and users that access it.
  3. 如果实体有吞吐量和查询性能的要求,请根据该实体使用功能分区。If an entity has throughput and query performance requirements, use functional partitioning based on that entity. 如果这样还是无法满足要求,请同时应用水平分区。If this still doesn't satisfy the requirements, apply horizontal partitioning as well. 在大多数情况下,单个分区策略就足够了,但在某些情况下,结合两种策略会更有效。In most cases, a single partitioning strategy will suffice, but in some cases it is more efficient to combine both strategies.

  4. 考虑跨分区并行运行查询以改善性能。Consider running queries in parallel across partitions to improve performance.

针对可用性设计分区Designing partitions for availability

将数据分区可以确保整个数据集不会构成单一故障点,并可确保数据集的单个子集可以独立进行管理,从而提高应用程序的可用性。Partitioning data can improve the availability of applications by ensuring that the entire dataset does not constitute a single point of failure and that individual subsets of the dataset can be managed independently.

请注意以下会影响可用性的因素:Consider the following factors that affect availability:

数据对业务运营的重要程度How critical the data is to business operations. 确定属于重要业务信息的数据(例如交易),以及较不重要的操作数据(例如日志文件)。Identify which data is critical business information, such as transactions, and which data is less critical operational data, such as log files.

  • 请考虑使用适当的备份计划将重要数据存储在高度可用的分区中。Consider storing critical data in highly available partitions with an appropriate backup plan.

  • 为不同的数据集建立独立的管理和监视过程。Establish separate management and monitoring procedures for the different datasets.

  • 将具有相同级别重要性的数据放在相同的分区中,以便可以按照相应的频率一同备份。Place data that has the same level of criticality in the same partition so that it can be backed up together at an appropriate frequency. 例如,保存交易数据的分区需要备份的频率可能高于保存日志记录或跟踪信息的分区。For example, partitions that hold transaction data might need to be backed up more frequently than partitions that hold logging or trace information.

单个分区的管理方式How individual partitions can be managed. 将分区设计为支持单独管理和维护可提供多种优势。Designing partitions to support independent management and maintenance provides several advantages. 例如:For example:

  • 如果分区故障,可以独立恢复该分区,不需使用在其他分区中访问数据的应用程序。If a partition fails, it can be recovered independently without applications that access data in other partitions.

  • 按地理区域将数据分区可以在每个位置的非高峰时段进行计划的维护任务。Partitioning data by geographical area allows scheduled maintenance tasks to occur at off-peak hours for each location. 确保分区不太大,以防在此期间完成任何计划内维护。Ensure that partitions are not too large to prevent any planned maintenance from being completed during this period.

是否要跨分区复制重要数据Whether to replicate critical data across partitions. 此策略可以提高可用性和性能,但也可能会造成一致性问题。This strategy can improve availability and performance, but can also introduce consistency issues. 在每个副本中同步所做的更改需要时间。It takes time to synchronize changes with every replica. 在这段时间,不同的分区会包含不同的数据值。During this period, different partitions will contain different data values.

应用程序设计注意事项Application design considerations

分区会增大系统设计和开发的复杂性。Partitioning adds complexity to the design and development of your system. 即使系统一开始只包含单个分区,也必须将分区视为系统设计的基础部分。Consider partitioning as a fundamental part of system design even if the system initially only contains a single partition. 如果事后才解决分区问题,则会更为困难,因为有一个实时系统需要维护:If you address partitioning as an afterthought, it will be more challenging because you already have a live system to maintain:

  • 数据访问逻辑需要修改。Data access logic will need to be modified.
  • 可能需要迁移大量现有数据以跨分区分布数据。Large quantities of existing data may need to be migrated, to distribute it across partitions.
  • 用户期望在迁移期间能够继续使用系统。Users expect to be able to continue using the system during the migration.

在某些情况下,分区并不重要,因为初始数据集很小,可以轻松地由单个服务器处理。In some cases, partitioning is not considered important because the initial dataset is small and can be easily handled by a single server. 对某些工作负荷来说,这可能是真的,但是许多商务系统需要随用户数的增加而扩展。This might be true for some workloads, but many commercial systems need to expand as the number of users increases.

另外,并非只有大型数据存储才受益于分区。Moreover, it's not only large data stores that benefit from partitioning. 例如,数百个并发客户端可能会重度访问一个小型数据存储。For example, a small data store might be heavily accessed by hundreds of concurrent clients. 在此情况下,将数据分区有助于减少争用并提高吞吐量。Partitioning the data in this situation can help to reduce contention and improve throughput.

在设计数据分区方案时,请注意以下几点:Consider the following points when you design a data partitioning scheme:

尽量减少跨分区数据访问操作Minimize cross-partition data access operations. 尽可能将每个分区中最常见数据库操作的数据保存在一起,尽量减少跨分区数据访问操作。Where possible, keep data for the most common database operations together in each partition to minimize cross-partition data access operations. 跨分区查询可能比在单个分区中查询更费时,而优化一个查询集的分区可能对其他查询集造成不利影响。Querying across partitions can be more time-consuming than querying within a single partition, but optimizing partitions for one set of queries might adversely affect other sets of queries. 如果必须跨分区查询,则应通过运行并行查询并在应用程序中聚合结果来尽量减少查询时间。If you must query across partitions, minimize query time by running parallel queries and aggregating the results within the application. (在某些情况下可能无法使用这种方法,例如一个查询的结果会在下一个查询中使用时。)(This approach might not be possible in some cases, such as when the result from one query is used in the next query.)

考虑复制静态引用数据。Consider replicating static reference data. 如果查询使用相对静态的引用数据,例如邮政编码表或产品列表,请考虑将该数据复制到所有分区,以减少不同分区中的独立查找操作。If queries use relatively static reference data, such as postal code tables or product lists, consider replicating this data in all of the partitions to reduce separate lookup operations in different partitions. 这种方法还可以降低引用数据变成“热”数据集(来自整个系统的流量很大)的可能性。This approach can also reduce the likelihood of the reference data becoming a "hot" dataset, with heavy traffic from across the entire system. 不过,将更改同步到引用数据会产生额外的开销。However, there is an additional cost associated with synchronizing any changes to the reference data.

尽量减少跨分区联接。Minimize cross-partition joins. 尽量减少跨垂直和功能分区的引用完整性的要求。Where possible, minimize requirements for referential integrity across vertical and functional partitions. 在这些方案中,应用程序负责维护跨分区的引用完整性。In these schemes, the application is responsible for maintaining referential integrity across partitions. 跨多个分区联接数据的查询效率不高,因为应用程序通常需要按照先键后外键的方式执行连续查询。Queries that join data across multiple partitions are inefficient because the application typically needs to perform consecutive queries based on a key and then a foreign key. 建议复制或取消规范化相关的数据。Instead, consider replicating or de-normalizing the relevant data. 如果需要进行跨分区联接,请对各分区运行并行查询,并在应用程序内部联接数据。If cross-partition joins are necessary, run parallel queries over the partitions and join the data within the application.

实现最终一致性Embrace eventual consistency. 评估强一致性是否为实际要求。Evaluate whether strong consistency is actually a requirement. 分布式系统中的一种常见方法是实施最终一致性。A common approach in distributed systems is to implement eventual consistency. 每个分区中的数据将单独更新,应用程序逻辑可确保所有更新成功完成。The data in each partition is updated separately, and the application logic ensures that the updates are all completed successfully. 此外,它还会处理在运行最终一致操作时查询数据所造成的不一致。It also handles the inconsistencies that can arise from querying data while an eventually consistent operation is running.

考虑查询如何查找正确的分区Consider how queries locate the correct partition. 如果查询必须扫描所有分区来查找所需的数据,即使运行多个并行查询,也会对性能产生严重的影响。If a query must scan all partitions to locate the required data, there is a significant impact on performance, even when multiple parallel queries are running. 使用垂直和功能分区时,查询可以自然地指定分区。With vertical and functional partitioning, queries can naturally specify the partition. 另一方面,水平分区可能会导致项的查找变得困难,因为每个分片都有相同的架构。Horizontal partitioning, on the other hand, can make locating an item difficult, because every shard has the same schema. 典型的解决方案是保留一种映射,用于查找特定项的分片位置。A typical solution to maintain a map that is used to look up the shard location for specific items. 此映射可以在应用程序的分片逻辑中实施,或者由数据存储维护(如果数据存储支持透明分片)。This map can be implemented in the sharding logic of the application, or maintained by the data store if it supports transparent sharding.

考虑定期重新平衡分片Consider periodically rebalancing shards. 使用水平分区时,重新平衡分片可能有助于根据大小和工作负荷均衡地分配数据,尽量减少热点,最大化查询性能,并解决物理存储限制。With horizontal partitioning, rebalancing shards can help distribute the data evenly by size and by workload to minimize hotspots, maximize query performance, and work around physical storage limitations. 不过,这是一个复杂的任务,通常需要使用定制工具或过程。However, this is a complex task that often requires the use of a custom tool or process.

复制分区。Replicate partitions. 复制每个分区可以进一步防范故障。If you replicate each partition, it provides additional protection against failure. 如果单个副本发生故障,查询可以定向到工作副本。If a single replica fails, queries can be directed toward a working copy.

如果达到了分区策略的物理限制,可能需要将缩放性扩展到其他级别If you reach the physical limits of a partitioning strategy, you might need to extend the scalability to a different level. 例如,如果分区位于数据库级别,则可能需要在多个数据库中查找或复制分区。For example, if partitioning is at the database level, you might need to locate or replicate partitions in multiple databases. 如果分区已在数据库级别,而物理限制成为一个问题,则可能需要在多个托管帐户中查找或复制分区。If partitioning is already at the database level, and physical limitations are an issue, it might mean that you need to locate or replicate partitions in multiple hosting accounts.

避免执行在多个分区中访问数据的事务Avoid transactions that access data in multiple partitions. 某些数据存储针对修改数据的操作实施事务一致性和完整性,但仅当数据位于单个分区时才能如此。Some data stores implement transactional consistency and integrity for operations that modify data, but only when the data is located in a single partition. 如果需要跨多个分区的事务支持,可能需要实施此支持作为应用程序逻辑的一部分,因为大多数分区系统不提供本机支持。If you need transactional support across multiple partitions, you will probably need to implement this as part of your application logic because most partitioning systems do not provide native support.

所有数据存储都需要某种操作管理和监视活动。All data stores require some operational management and monitoring activity. 任务的范围可能包括加载数据、备份和还原数据、重新组织数据,以及确保系统正常有效地执行。The tasks can range from loading data, backing up and restoring data, reorganizing data, and ensuring that the system is performing correctly and efficiently.

请注意以下会影响操作管理的因素:Consider the following factors that affect operational management:

  • 将数据分区时如何实施适当的管理和操作任务How to implement appropriate management and operational tasks when the data is partitioned. 这些任务可能包括备份与还原、存档数据,监视系统及其他管理任务。These tasks might include backup and restore, archiving data, monitoring the system, and other administrative tasks. 例如,在备份和还原操作期间保持逻辑一致性可能是一个难题。For example, maintaining logical consistency during backup and restore operations can be a challenge.

  • 如何将数据载入多个分区,以及如何添加来自其他源的新数据How to load the data into multiple partitions and add new data that's arriving from other sources. 某些工具和实用程序可能不支持分片数据操作(例如将数据载入正确的分区)。Some tools and utilities might not support sharded data operations such as loading data into the correct partition.

  • 如何定期存档和删除数据How to archive and delete the data on a regular basis. 若要防止分区过多增长,需要定期存档和删除数据 (例如每月) 。To prevent the excessive growth of partitions, you need to archive and delete data on a regular basis (such as monthly). 可能需要转换数据以符合不同的存档架构。It might be necessary to transform the data to match a different archive schema.

  • 如何找出数据完整性问题How to locate data integrity issues. 考虑运行定期过程来查找数据完整性问题,例如一个分区的数据引用了另一个分区中缺少的信息。Consider running a periodic process to locate any data integrity issues, such as data in one partition that references missing information in another. 该过程可能会尝试自动修复这些问题,或者为手动查看生成报告。The process can either attempt to fix these issues automatically or generate a report for manual review.

重新平衡分区Rebalancing partitions

随着系统成熟,可能需要调整分区方案。As a system matures, you might have to adjust the partitioning scheme. 例如,各个分区可能会开始获得一个不相称的流量,并使其处于热状态,导致过度争用。For example, individual partitions might start getting a disproportionate volume of traffic and become hot, leading to excessive contention. 或者,你可能低估了某些分区中的数据量,导致这些分区达到容量限制。Or you might have underestimated the volume of data in some partitions, causing some partitions to approach capacity limits.

某些数据存储(例如 Cosmos DB)可能会自动对分区进行重新平衡。Some data stores, such as Cosmos DB, can automatically rebalance partitions. 在其他情况下,重新平衡是由两个阶段组成的管理任务:In other cases, rebalancing is an administrative task that consists of two stages:

  1. 确定新的分区策略。Determine a new partitioning strategy.

    • 哪些分区需要拆分(或者可能需要合并)?Which partitions need to be split (or possibly combined)?
    • 新分区键是什么?What is the new partition key?
  2. 将数据从旧的分区方案迁移到一组新的分区。Migrate data from the old partitioning scheme to the new set of partitions.

也许能够在分区被使用时在分区之间迁移数据,具体取决于数据存储。Depending on the data store, you might be able to migrate data between partitions while they are in use. 这称为联机迁移。This is called online migration. 如果这不可行,可能需要在重定位数据时使分区不可用(脱机迁移)。If that's not possible, you might need to make partitions unavailable while the data is relocated (offline migration).

脱机迁移Offline migration

脱机迁移通常更简单,因为这样可以减少发生争用的可能性。Offline migration is typically simpler because it reduces the chances of contention occurring. 从概念上讲,脱机迁移的工作方式如下所示:Conceptually, offline migration works as follows:

  1. 将分区标记为脱机。Mark the partition offline.
  2. 拆分/合并数据,并将其转移到新分区。Split-merge and move the data to the new partitions.
  3. 验证数据。Verify the data.
  4. 使新分区联机。Bring the new partitions online.
  5. 删除旧分区。Remove the old partition.

可以选择在步骤 1 中将分区标记为只读,使应用程序仍然能够在移动数据时读取数据。Optionally, you can mark a partition as read-only in step 1, so that applications can still read the data while it is being moved.

联机迁移Online migration

联机迁移执行起来更复杂,但对用户的干扰较少。Online migration is more complex to perform but less disruptive. 该过程与脱机迁移类似,不同之处在于,原始分区不会标记为脱机。The process is similar to offline migration, except the original partition is not marked offline. 根据迁移过程的粒度(例如,是逐项迁移还是逐分片迁移),客户端应用程序中的数据访问代码可能需要处理保存在两个位置(原始分区和新分区)的数据的读取和写入。Depending on the granularity of the migration process (for example, item by item versus shard by shard), the data access code in the client applications might have to handle reading and writing data that's held in two locations, the original partition and the new partition.

以下设计模式可能与方案相关:The following design patterns might be relevant to your scenario:

  • 分配模式介绍了一些进行数据分片的常见策略。The sharding pattern describes some common strategies for sharding data.

  • 索引表模式介绍了如何基于数据创建辅助索引。The index table pattern shows how to create secondary indexes over data. 此方法可让应用程序使用未引用集合主键的查询快速检索数据。An application can quickly retrieve data with this approach, by using queries that do not reference the primary key of a collection.

  • 具体化视图模式介绍了如何生成预填充的视图,这些视图汇总数据以支持快速查询操作。The materialized view pattern describes how to generate prepopulated views that summarize data to support fast query operations. 如果包含汇总数据的分区分布在多个站点上,此方法可能对分区的数据存储很有用。This approach can be useful in a partitioned data store if the partitions that contain the data being summarized are distributed across multiple sites.

后续步骤Next steps