Zookeeper middleware solution
Middleware ZooKeeper (distributed service framework)
1 Introduction
Zookeeper is a software that provides consistency services for distributed applications. It is an open source distributed coordination service and a sub-project of the open source hadoop project. It can provide configuration information management, naming, distributed synchronization, cluster management, database switching, etc. service. It is not suitable for storing large amounts of information, but can be used to store small amounts of information such as configuration, publishing, and subscription. Hadoop, Storm, message middleware, RPC service framework, distributed database synchronization system, these are the application scenarios of Zookeeper.
Zookeeper can act as a service registry (Service Registry), allowing multiple service providers to form a cluster, allowing service consumers to obtain specific service access addresses (ip+ports) through the service registry to access specific service providers.
Official: ZooKeeper is an open source distributed service framework. It is a sub-project of the Apache Hadoop project. It is mainly used to solve some problems in distributed application scenarios, such as: unified naming service, state synchronization service, cluster management, Distributed application configuration management, etc., it supports Standalone mode and distributed mode. In distributed mode, it can provide high-performance and reliable coordination services for distributed applications, and the use of ZooKeeper can greatly simplify the implementation of distributed coordination services. The development of distributed applications greatly reduces costs.
As a distributed service framework, Zookeeper is mainly used to solve the consistency problem of application systems in distributed clusters. It can provide data storage based on a directory node tree similar to the file system, but Zookeeper is not used to store data specifically Yes, its function is mainly used to maintain and monitor the status changes of your stored data. By monitoring the changes of these data states, data-based cluster management can be achieved.
The goal of ZooKeeper is to encapsulate key error-prone services and provide users with simple and easy-to-use interfaces and systems with high performance and stable functions.
Zookeeper is a distributed service management framework designed based on the observer pattern, responsible for storing and managing related data, and receiving the registration of observers. Once the state of these data changes, zookeeper is responsible for notifying those observers who have registered in the zookeeper cluster and are concerned about the changes in these states, so that the observers can perform related operations.
2. Three roles of Zookeeper:
1) Leader 2) Follower 3) Observer
A ZooKeeper cluster has only one Leader at a time, and the others are Followers or Observers.
3. Division of node read and write services:
1) All machines in the ZooKeeper cluster select one called Leader through a Leader election process
The Leader server provides read and write services for clients.
2) Both Follower and Observer can provide read services, but cannot provide write services. The only difference between the two is that
The Observer machine does not participate in the Leader election process, nor does it participate in the "over half write success" strategy of write operations. Therefore, the Observer can improve the read performance of the cluster without affecting the write performance.
The default service port of ZooKeeper is 2181.
4. The structure of zookeeper is actually a tree structure, the leader is equivalent to the root node, and other nodes are equivalent to follow nodes, and each node retains its own content.
The number of nodes in the Zookeeper cluster is generally an odd number (>=3). If the master in the cluster hangs up and the number of remaining nodes is more than half, a new master node can be recommended to continue to provide services to the outside world.
The client initiates a transaction request, and the result of the transaction request is consistent across all machines in the Zookeeper cluster. There will not be a situation where some machines in the cluster have applied the transaction, while there are cases where the machine in another part of the cluster does not apply the transaction. On any machine in the Zookeeper cluster, the data model of the server it sees is the same. Zookeeper can guarantee the order of client requests. Each request is assigned a globally unique incremental number to reflect the order of transaction operations. Zookeeper saves the entire amount of data in memory and directly serves all non-transactional requests. Its performance is very prominent in scenarios where read operations are the mainstay.
The data structure used by Zookeeper is a tree structure, and the root node is "/". The nodes in the Zookeeper cluster are divided into leader, follower, and observer according to their identity characteristics. The leader is responsible for client writer type requests; follower is responsible for client reader type requests and participates in leader election; observer is a special follower that can receive client reader requests, but will not participate in elections, and can be used to expand system support capabilities. Improve reading speed.
5. Zookeeper uses the ZAB atomic message broadcast protocol, and the consensus algorithm between nodes is Paxos, which can ensure the consistency of data in a distributed environment. High availability in a distributed scenario is a feature of Zookeeper, which can be implemented by a third-party client, namely the Curator framework.
6. Zookeeper cluster management
Cluster management: whether there are machines withdrawing and joining, and master election.
For the first point, all machines agree to create a temporary directory node under the parent directory GroupMembers, and then monitor the child node change messages of the parent directory node. Once a machine hangs up, the machine is disconnected from zookeeper, the temporary directory node it created is deleted, and all other machines are notified that a brother directory has been deleted.
The addition of new machines is similar. All machines receive notifications: the new brother directory is added, and highcount is there again. For the second point, we change it slightly. All machines create temporary sequential numbered directory nodes, and each time the machine with the smallest number is selected as the master Just fine.
7. Working principle of Zookeeper
The core of Zookeeper is atomic broadcasting. This mechanism ensures synchronization between servers. The protocol that implements this mechanism is called the Zab protocol. The Zab protocol has two modes, which are recovery mode (primary selection) and broadcast mode (synchronization). When the service is started or after the leader crashes, Zab enters the recovery mode. When the leader is elected and most of the servers are synchronized with the leader's state, the recovery mode ends. State synchronization ensures that the leader and server have the same system state.
From the perspective of design patterns, Zookeeper is a distributed service management framework designed based on the observer pattern. It is responsible for storing and managing the data that everyone cares about, and then accepts the observer's registration. Once the state of these data changes, Zookeeper will Will be responsible for notifying those observers who have registered on Zookeeper to respond accordingly, so as to achieve a similar Master/Slave management model in the cluster.
Zookeeper
Data model
Tree structure, the root node is /, each layer is divided by /, and only absolute paths can be used for access. This is because most of the zookeeper scenarios are to locate nodes on the data model, that is, to find nodes, so the bottom layer uses a hash structure and uses the full path of the node as the key to improve the search performance.
Permanent node: Permanently exists after creation, and it is displayed that delete is deleted by calling delete.
Temporary node: After the client that created the temporary node closes the connection (timeout or abnormality is also counted), it is automatically deleted.
Ordered nodes: When creating permanent or temporary nodes, zookeeper automatically appends a monotonically increasing serial number after the node name to indicate the order of node creation.
The data maintained by the node includes:
Node data, use binary array byte[] to store
ACL access control information
Child node data, temporary absolute points are not allowed to have child nodes, so the child node field is null
The field stat that records its own status information, see the figure below for specific information
Pessimistic lock
It is agreed that /lock is the lock node, and the client that successfully creates the /lock node obtains the lock.
Optimistic lock
The use of CAS is realized by the version information stored in the node.
Watch mechanism
A subscription monitoring mechanism, which can be used by passing the Watcher time parameter to the client's construction method, getData, exists and getChildren method parameters.
new ZooKeeper(String connectString, int sessionTimeout, Watcher watcher);getData(String path, Watcher watcher, Stat stat);
When the client connects to the server, it can monitor the creation, deletion, data change, and update of sub-nodes of data nodes.
The client marks the message as a watch message and sends it to the server. After receiving it, the server records the watch in the watchManager and returns a response. After the client receives the response, it records the watch in its own zkwatchManager.
When the server starts the watch, it will call back the client according to the record in watchmanage, and the client will call back the response function according to zkWatchManager.
Note that watch is one-time and will be deleted after the callback. Repeat registration if necessary.
Watch can be used to implement publish and subscribe functions such as configuration center.
上一篇：無(wú)
