Created on 10-13-2015 02:42 PM
It's coming up more often now, the need to "add" host to an existing cluster without using the Ambari UI. I once thought the Blueprints were only useful to "initially" provision a cluster. But they're also quite helpful to extend your cluster as well.
In the following example, we'll use a new feature in Ambari 2.1, called "Auto-Discovery" to provision the first node in our cluster.
Do NOT manually register the Host with Ambari Server yet. We're going to demonstrate "Auto-Discovery" to initialize new hosts.
Check that NO hosts have been registered with the cluster.
Blueprint has be registered with Ambari< "Blueprints": < "stack_name": "HDP", "stack_version": "2.3" >, "configurations": [], "host_groups": [ < "cardinality": "1", "components": [ < "name": "ZOOKEEPER_SERVER" >, < "name": "ZOOKEEPER_CLIENT" >], "configurations": [], "name": "zookeeper_host_group" >, < "cardinality": "1", "components": [ < "name": "KAFKA_BROKER" >], "configurations": [], "name": "kafka_host_group" > ] >
Save this above to a file and register the Blueprint with you cluster.
Create your cluster
Now lets create our cluster and use the Auto-Discovery feature to provision the host(s). This is the 'template' file we use to provision the cluster.
At this point the cluster has been created in Ambari, but you should NOT have any hosts. Ambari Server is sitting around waiting for a host to be registered with the cluster matching the "predicate" above. The "host_count" is also an important part of the criteria. The process will not start until the number of host in the "host_count" are available. For our demonstration here, we've set it to 1. In the real world, you would need 3 to create a valid ZooKeeper quorum.
Manually Register a Host (just one, for now)
Install an Ambari Agent on your host and configure it to talk to Ambari. The host should match the predicate above "cpu_count>0", which would be just about everything. But you get the point.
Start the Ambari-Agent and return back to the Ambari Server UI. In short order, you should see operations kicking off to provision the node and add the service. Now you have a cluster with 1 node and a running ZooKeeper Service.
With a second host, lets manually register the agent with Ambari. Check via the API that the host has been registered with the Ambari Server
Notice that the previously registered Blueprint "basic1", has a host group for Kafka Brokers. Let's add this newly registered host to our cluster as a Kakfa Broker.
Create a file to contain the message body and save it.
Add and provision the host with the services specified in the Blueprint.
curl -i -H "X-Requested-By: ambari" -u admin:xxxx -X POST -d @kafka_host.json /api/v1/clusters//hosts/" target="_blank" rel="nofollow noopener noreferrer">http:///api/v1/clusters//hosts/ ;
The is the name that it's been registered to Ambari by, should be the host FQDN. Double check the name returned by the hosts query above.
This will add the host to you new cluster and provision it with the services configured in the Blueprint.
In the past, Blueprints were used to initialized clusters. Now you can use them to "extend" your cluster. Avoid trying to reverse engineer the process of installing and configuring services through the REST API, use "Blueprints". You'll save yourself a lot of effort!!
Created on 07-04-2016 05:08 PM
It’s an interesting fact that a huge data can be saved easily on a secure cloud space, and that is why we are able to operate e-commerce websites in a risk-free environment. However, the first step is to buy a relevant domain name, and then get a reliable web hosting platform that can handle heavy load with utter smoothness. I follow the suggestions from WebHostingGeeks.com whenever I create new websites for my clients.
Created on 11-10-2016 09:28 AM
I have created a cluster using ambari UI.so Is it possible to scale it through Ambari Blueprint API?
When I am trying to scale (adding one node with services) it is showing the following message:
"message" : "Topology validation failed: org.apache.ambari.server.topology.InvalidTopologyException: Unable to retrieve cluster topology for cluster. This is most likely a result of trying to scale a cluster via the API which was created using the Ambari UI. At this time only clusters created via the API using a blueprint can be scaled with this API.
Please suggest me whether its possible or not to add nodes using API if my cluster was previously created by ambari UI?
Created on 05-18-2018 01:26 PM
I have been trying your suggestion, added the host via the above API call and get the following:
18 May 2018 14:18:37,613 INFO [qtp-ambari-agent-326] TopologyManager:637 - TopologyManager.onHostRegistered: Entering 18 May 2018 14:18:37,613 INFO [qtp-ambari-agent-326] TopologyManager:698 - TopologyManager: Queueing available host node5 18 May 2018 14:19:37,959 WARN [alert-event-bus-1] AlertReceivedListener:497 - Unable to process alert ambari_agent_disk_usage for cluster idp and host node5 because the host is not a part of the cluster. 18 May 2018 14:20:06,779 INFO [ambari-client-thread-74] TopologyManager:485 - TopologyManager.scaleHosts: Entering 18 May 2018 14:20:06,779 INFO [ambari-client-thread-74] ClusterTopologyImpl:158 - ClusterTopologyImpl.addHostTopology: added host = node5 to host group = host_group_1 18 May 2018 14:20:06,780 INFO [ambari-client-thread-74] HostRequest:205 - Skipping Start task creation since provision action = INSTALL_ONLY 18 May 2018 14:20:06,782 INFO [ambari-client-thread-74] HostRequest:244 - Skipping create of START task for KAFKA_BROKER on PENDING HOST ASSIGNMENT : HOSTGROUP=host_group_1. 18 May 2018 14:20:06,783 INFO [ambari-client-thread-74] HostRequest:244 - Skipping create of START task for ZOOKEEPER_SERVER on PENDING HOST ASSIGNMENT : HOSTGROUP=host_group_1. 18 May 2018 14:20:06,784 INFO [ambari-client-thread-74] HostRequest:244 - Skipping create of START task for METRICS_MONITOR on PENDING HOST ASSIGNMENT : HOSTGROUP=host_group_1. 18 May 2018 14:20:06,784 INFO [ambari-client-thread-74] HostRequest:244 - Skipping create of START task for NIFI_MASTER on PENDING HOST ASSIGNMENT : HOSTGROUP=host_group_1. 18 May 2018 14:20:06,785 INFO [ambari-client-thread-74] HostRequest:244 - Skipping create of START task for LOGSEARCH_LOGFEEDER on PENDING HOST ASSIGNMENT : HOSTGROUP=host_group_1. 18 May 2018 14:20:06,786 INFO [ambari-client-thread-74] HostRequest:244 - Skipping create of START task for ZOOKEEPER_CLIENT on PENDING HOST ASSIGNMENT : HOSTGROUP=host_group_1. 18 May 2018 14:20:06,786 INFO [ambari-client-thread-74] HostRequest:99 - HostRequest: Created request for host: node5 18 May 2018 14:20:06,786 INFO [ambari-client-thread-74] LogicalRequest:437 - LogicalRequest.createHostRequests: all host requests size 1 , outstanding requests size = 0 18 May 2018 14:20:06,790 INFO [ambari-client-thread-74] TopologyManager:923 - TopologyManager.createLogicalRequest: created LogicalRequest with and completed persistence of this request. 18 May 2018 14:20:06,793 INFO [ambari-client-thread-74] TopologyManager:845 - TopologyManager.processRequest: Entering 18 May 2018 14:20:06,794 INFO [ambari-client-thread-74] TopologyManager:863 - TopologyManager.processRequest: host name = node5 is mapped to LogicalRequest and will be removed from the reserved hosts. 18 May 2018 14:20:06,794 INFO [ambari-client-thread-74] TopologyManager:876 - TopologyManager.processRequest: offering host name = node5 to LogicalRequest May 2018 14:20:06,794 INFO [ambari-client-thread-74] LogicalRequest:101 - LogicalRequest.offer: attempting to match a request to a request for a reserved host to hostname = node5 18 May 2018 14:20:06,794 INFO [ambari-client-thread-74] LogicalRequest:110 - LogicalRequest.offer: request mapping ACCEPTED for host = node5 18 May 2018 14:20:06,794 INFO [ambari-client-thread-74] LogicalRequest:113 - LogicalRequest.offer returning response, reservedHost list size = 0 18 May 2018 14:20:06,795 INFO [ambari-client-thread-74] TopologyManager:886 - TopologyManager.processRequest: host name = node5 was ACCEPTED by LogicalRequest , host has been removed from available hosts. 18 May 2018 14:20:06,795 INFO [ambari-client-thread-74] ClusterTopologyImpl:158 - ClusterTopologyImpl.addHostTopology: added host = node5 to host group = host_group_1 18 May 2018 14:20:06,797 INFO [ambari-client-thread-74] TopologyManager:963 - TopologyManager.processAcceptedHostOffer: queue tasks for host = node5 which responded ACCEPTED 18 May 2018 14:20:06,797 INFO [ambari-client-thread-74] TopologyManager:988 - TopologyManager.processAcceptedHostOffer: queueing tasks for host = node5 18 May 2018 14:20:06,797 INFO [ambari-client-thread-74] TopologyManager:904 - TopologyManager.processRequest: not all required hosts have been matched, so adding LogicalRequest to outstanding requests
What can I do to move the request from PENDING to actually started?