Redhat Cluster - Interview Questions and Answers: Part 1



1. What is CMAN
  • Basically, cluster manager is a component of the cluster project that handles communications between nodes in the cluster.
  • CMAN is Cluster Manager. It manages cluster quorum and cluster membership.
  • CMAN runs on each node of a cluster

2. What is RGManager

RGManager manages and provides failover capabilities for collections of cluster resources called services, resource groups, or resource trees.

In the event of a node failure, RGManager will relocate the clustered service to another node with minimal service disruption. You can also restrict services to certain nodes, such as restricting  httpd to one group of nodes while  mysql can be restricted to a separate set of nodes.

When the cluster membership changes, openais tells the cluster that it needs to recheck it’s resources. This causes rgmanager, the resource group manager, to run. It will examine what changed and then will start, stop, migrate or recover cluster resources as needed.
Within rgmanager, one or more resources are brought together as a service. This service is then optionally assigned to a failover domain, an subset of nodes that can have preferential ordering.


3. What is Cluster Quorum


  • Quorum is a voting algorithm used by CMAN.
  • CMAN keeps a track of cluster quorum by monitoring the count of number of nodes in cluster.
  • If more than half of members of a cluster are in active state, the cluster is said to be in Quorum
  • If half or less than half of the members are not active, the cluster is said to be down and all cluster activities will be stopped
  • Quorum is defined as the minimum set of hosts required in order to provide service and is used to prevent split-brain situations.
  • The quorum algorithm used by the RHCS cluster is called “simple majority quorum”, which means that more than half of the hosts must be online and communicating 
  • in order to provide service.
4.  What is split-brain

It is a condition where two instances of the same cluster are running and trying to access same resource at the same time, resulting in corrupted cluster integrity
Cluster must maintain quorum to prevent split-brain issues

It's necessary for a cluster to maintain quorum to prevent 'split-brain' problems. If we didn't enforce quorum, a communication error on that same thirteen-node cluster
may cause a situation where six nodes are operating on the shared disk, and another six were also operating on it, independently. Because of the communication error,
the two partial-clusters would overwrite areas of the disk and corrupt the file system. With quorum rules enforced, only one of the partial clusters can use the shared storage, thus protecting data integrity.

Quorum doesn't prevent split-brain situations, but it does decide who is dominant and allowed to function in the cluster. Should split-brain occur, quorum prevents more than one cluster group from doing anything.


5. What is Fencing


  • Fencing is the disconnection of a node from the cluster’s shared storage. Fencing cuts off I/O from shared storage, thus ensuring data integrity. The cluster infrastructure performs fencing through the fence daemon,  fenced.
  • Power fencing — A fencing method that uses a power controller to power off an inoperable node.
  • storage fencing — A fencing method that disables the Fibre Channel port that connects storage to an inoperable node.
  • Other fencing — Several other fencing methods that disable I/O or power of an inoperable node, including IBM Bladecenters, PAP, DRAC/MC, HP ILO, IPMI, IBM RSA II, and others.

6. What is Quorum disk


  • In case of a 2 node cluster, quorum disk acts as a tie-breaker and prevents split-brain issue
  • If a node has access to network and quorum disk, it is active
  • If a node has lost access to network or quorum disk, it is inactive and can be fenced
  • A Quorum disk, known as a qdisk is small partition on SAN storage used to enhance quorum. It generally carries enough votes to allow even a single node to take quorum during a cluster partition. 
  • It does this by using configured heuristics, that is custom tests, to decided which which node or partition is best suited for providing clustered services during a cluster reconfiguration.

7. How to set up a quorum disk/partition?

Note that if you configure a quorum disk/partition, you don't want two_node="1" or expected_votes="2" since the quorum disk solves the voting imbalance. 

You want two_node="0" and expected_votes="3" (or nodes + 1 if it's not a two-node cluster). However, since 0 is the default value for two_node, you don't need to specify it at all. 

If this is an existing two-node cluster and you're changing the two_node value from "1" to "0", you'll have to stop the entire cluster and restart it after the configuration is changed (normally, the cluster doesn't have to be stopped and restarted for configuration changes, but two_node is a special case.) Basically, you want something like this in your /etc/cluster/cluster.conf:

  <cman two_node="0" expected_votes="3" .../>
    <clusternodes>
       <clusternode name="node1" votes="1" .../>
       <clusternode name="node2" votes="1" .../>
    </clusternodes>
  <quorumd device="/dev/mapper/lun01" votes="1"/>

Note: You don't have to use a disk or partition to prevent two-node fence-cycles; you can also set your cluster up this way. 

You can set up a number of different heuristics for the qdisk daemon. For example, you can set up a redundant NIC with a crossover cable and use ping operations to the local router/switch to break the tie (this is typical, actually, and is called an IP tie breaker). 
A heuristic can be made to check anything, as long as it is a shared resource.

8 .What can cause a node to leave the cluster?

A node may leave the cluster for many reasons. Among them:


  1. Shutdown: cman_tool leave was run on this node
  2. Killed by another node. The node was killed with either by cman_tool kill or qdisk.
  3. Panic: cman failed to allocate memory for a critical data structure or some other very bad internal failure.
  4. Removed: Like 1, but the remainder of the cluster can adjust quorum downwards to keep working.
  5. Membership Rejected: The node attempted to join a cluster but it's
  6. cluster.conf file did not match that of the other nodes. To find the real reason for this you need to examine the syslog of all the valid cluster members to find out why it was rejected.
  7. Inconsistent cluster view: This is usually indicative of a bug but it can also happen if the network is extremely unreliable.
  8. Missed too many heartbeats: This means what it says. All nodes are expected to broadcast a heartbeat every 5 seconds (by default). If none is received within

9 . How can I define a two-node cluster if a majority is needed to reach quorum?

We had to allow two-node clusters, so we made a special exception to the quorum rules. There is a special setting "two_node" in the /etc/cluster.conf file that looks like this:

<cman expected_votes="1" two_node="1"/>

This will allow one node to be considered enough to establish a quorum. Note that if you configure a quorum disk/partition, you don't want two_node="1".


10. What is the best two-node network & fencing configuration?

In a two node cluster (where you are using two_node="1" in the cluster configuration, and w/o QDisk), there are several considerations you need to be aware of:

If you are using per-node power management of any sort where the device is not shared between cluster nodes, it must be connected to the same network used by CMAN for cluster communication. Failure to do so can result in both nodes simultaneously fencing each other, leaving the entire cluster dead, or end up in a fence loop. Typically, this includes all integrated power management solutions (iLO, IPMI, RSA, ERA, IBM Blade Center, Egenera Blade Frame, Dell DRAC, etc.), but also includes remote power switches (APC, WTI) if the devices are not shared between the two nodes.

It is best to use power-type fencing. SAN or SCSI-reservation fencing might work, as long as it meets the above requirements. If it does not, you should consider using a quorum disk or partition

If you can not meet the above requirements, you can use quorum disk or partition.


Post a Comment

26 Comments

  1. Hi Buddy,

    I learnt so much in such little time about UNIX / LINUX Tips and Tricks... Even a toddler could become smart reading of your amazing articles.

    Early versions of Unix contained a development environment sufficient to recreate the entire system from source code. Is it possible to do so for the new released OS?

    I read multiple articles and watched many videos about how to use this tool - and was still confused! Your instructions were easy to understand and made the process simple.


    MuchasGracias,
    Ajeeth Kapoor

    ReplyDelete
  2. Hola,


    Fully agree on Cluster Interview Questions. We’re seeing a lot of projects tackle big complex problems but few seem to have taken into consideration and in particular reasons to adopt.

    When you execute a program on your UNIX system, the system creates a special environment for that program. This environment contains everything needed for the system to run the program as if no other program were running on the system.

    I think the Linux phenomenon is quite delightful, because it draws so strongly on the basis that Unix provided. Linux seems to be the among the healthiest of the direct Unix derivatives, though there are also the various BSD systems as well as the more official offerings from the workstation and mainframe manufacturers

    Anyways great write up, your efforts are much appreciated.

    Shukran,
    Kevin

    ReplyDelete
  3. Python Training, SVR Technologies offers Online & Corporate Classes with Free Live Demo, We cover practical sessions and all the modules in Python..

    Angular Training,
    AWS Training Online,
    Best Online Training,
    Devops Training,
    Machine Learning Training,
    Mulesoft Training,
    Online Training Institute,
    Python Training,
    Salesforce Training,
    SAP Training,
    Tableau Training,
    Tibco Training




    Article:

    Python Training, SVR Technologies offers Online & Corporate Classes with Free Live Demo, We cover practical sessions and all the modules in Python..

    About SVR Technologies:

    24/7 Online Training | Corporate Training, Videos, Job Support. SVR Technologies building careers for Software Engineers With online Training self-paced.


    Features:

    >> Live Instructor LED Classes
    >> Experienced Faculty
    >> Free Video materials
    >> 24/7 Support
    >> Flexible Timings
    >> Lowest Fee

    online python training

    ReplyDelete
  4. This is very good, that you decided to devote the post to this theme. I guess, many people have some questions on different problems.

    ReplyDelete
  5. I find the Linux phenomenon to be quite delightful because it builds so heavily on the UNIX foundation. You can get your assignment and essay completed at essay master UK in a relatively simple and basic manner.

    ReplyDelete
  6. It's great that you decided to devote an article to this topic. Many people, I'm sure, have questions about many issues.

    ReplyDelete
  7. That's exactly what I'd like to know. It says "1 serving of spaghetti," which implies that it only serves one person. However, two garlic bulbs for one meal seems excessive to me. I'm not sure right now; I'll just tweak it up a little and give it a shot.

    ReplyDelete
  8. When i was need the checking the teeth missing and replacing the teeth so i am going at the best dental clinic in karachi to the checking and washing the teeth.

    ReplyDelete
  9. This comment has been removed by the author.

    ReplyDelete
  10. The cluster interview questions are perfect. Many projects are attempting to solve complicated issues. So now we decided to resolve students questions regarding their academic. We've online business management research paper writing service, we handle all your responsibilities. If you need any kind of help avail our service now.

    ReplyDelete
  11. This is very good, that you decided to devote the post to this theme.

    ReplyDelete
  12. I find the Linux phenomenon to be quite delightful because it builds so heavily on the UNIX foundation. You can get your assignment and essay completed at do my assignment UK in a relatively simple and basic manner.

    ReplyDelete
  13. Interview Questions and Answers are essential for job seekers who are preparing for their next job interview. These questions and answers can provide insights into the type of questions that might be asked during an interview, allowing candidates to better prepare their responses. However, preparing for an interview can be time-consuming, and some candidates may feel overwhelmed with the process. That's where the write my essay for me cheap comes in handy. For instance, if a candidate is struggling to prepare for an interview, they can seek assistance from online essay writing services that offer affordable rates. These services can provide them with expertly crafted answers to potential interview questions, allowing them to be better prepared for their interview

    ReplyDelete
  14. to make sure that our writer’s work is completely unique and one of a kind. So whenever you need a help with a research paper, you can always count on us! Online persuasive essay Writer uk

    ReplyDelete
  15. companies to choose from, so why move with us? Our moving crew have moved thousands of families like yours and ensured that they received the best moving experience, stress-free professional academic writing services UK

    ReplyDelete
  16. As a resume writer boston with in-depth knowledge of cluster management, I understand the critical components involved in ensuring seamless communication and failover capabilities within a cluster. From the Cluster Manager (CMAN) responsible for handling communications between nodes and managing cluster quorum and membership, to the Resource Group Manager (RGManager) that provides failover capabilities for cluster resources, I am well-versed in the intricacies of these systems. My expertise extends to configuring services, managing resource groups, and optimizing failover domains to minimize service disruption. By showcasing my proficiency in cluster management, I can help professionals emphasize their ability to effectively maintain and optimize cluster environments in their resume.

    ReplyDelete
  17. When it comes to Redhat Cluster interview questions, it's essential to have a solid understanding of the concepts to excel. Utilizing "statistics assignment help UK" can provide valuable insights into the performance and reliability of clusters. Understanding statistical metrics can be crucial in assessing cluster efficiency and optimizing resource allocation. Seeking assistance from experts ensures you're well-prepared for the interview and equipped to handle real-world cluster challenges effectively.

    ReplyDelete
  18. Redhat Cluster - Interview Questions and Answers: Part 1.' will introduce you to the world of technology. Your blog strikes a balance between information and preparation, much like Relax CBD does. These tips act as a calming remedy for tech interview nerves. awaiting more components to enhance our technological trip. Thank you for bringing wellness and knowledge to the IT world! #CBDTechBalance #TechInterviewPrep

    ReplyDelete
  19. Great content! These Redhat Cluster interview questions and answers are a valuable resource for aspiring professionals! And for those juggling career growth and parenting, the mention of teething tubes adds a unique touch.

    ReplyDelete
  20. question even by using internet. Is it possible for you to share here the question instead of the question code. Well, I often use professional essay writing services

    ReplyDelete
  21. While Redhat Cluster interview questions are essential for IT professionals, don't forget the importance of continuous learning. Just like staying updated in the tech field, those aspiring to be real estate professionals can benefit from a comprehensive 99-hour pre-licensing real estate course. Both fields demand a commitment to learning and staying current in their respective domains to excel in their careers.

    ReplyDelete
  22. Redhat Cluster - Interview Questions and Answers: Part 1" is a valuable resource for IT professionals preparing for interviews, much like finding the "Best Pizza Deals" is a handy resource for those seeking budget-friendly dining options. Both aim to provide solutions – one for career advancement, and the other for a satisfying meal. It's all about equipping yourself with the right knowledge, whether in the tech world or the culinary realm.

    ReplyDelete
  23. When I change the books in this basket I always make a quick announcement. I do not give a book talk and instead let them explore the books on their own. online essay writers uk

    ReplyDelete
  24. Crafting a compelling narrative is an art, and with the expertise of Michael Levin ghostwriter extraordinaire, your story can unfold seamlessly. Just as a Redhat Cluster ensures reliability in the tech world, Levin's skillful penmanship guarantees a robust and engaging narrative. Explore the synergy between storytelling finesse and literary expertise with Michael Levin, the trusted name in the realm of ghostwriting.

    ReplyDelete
  25. We've worked hard over the years to make our best law assignment help service completely customer-focused. Being innovative and creative is one of our guiding concepts. We only use competent and experienced assignment writers who can write original papers. Every ordinary assignment coursework writing Help

    ReplyDelete