Why doesn't nodetool work on my Cassandra Cluster?

Question

I have never been able to get nodetool to operate correctly to give me information about my cluster. We have a simple 3-node cluster of Cassandra 3.11.1 nodes running under CentOS 7.3 -- using the gossip snitch and replication. We have a special place on the file system where we're putting the node configurations, but these are appropriately set with environment variables. A "generic" configuration is in the environment for client applications.

These are attempts at using nodetool (I have replaced the exact server name with /hostname/.)

  % nodetool -h /hostname/ status
  nodetool: Found unexpected parameters: [/hostname/, status]
  See 'nodetool help' or 'nodetool help '.

maybe the help is wrong and flags go at the end? Running this:

  % nodetool  status -h /hostname/

gives me usage information.

My favorite is this:

  % nodetool help status
  nodetool: For input string: "help"
  See 'nodetool help' or 'nodetool help '.

I can connect with cqlsh. I get the same response from nodetool whether I'm on a remote host or one of the cluster nodes. Python clients connect fine and we've been able to create keyspaces and bulk load data from remote hosts.

I am sure there is some configuration I have overlooked but I can't find any information about what it might be in the docs.

Any suggestions on how to get nodetool working?

Answer 1

After some tinkering I figured it out; posting here to close the topic and to help others.

All the scripts in the conf/ directory work in concert and there are a number of environment variables you need to redirect if you install in a non-standard place (i.e. a tarball installation).

For each node in our cluster, we're (now) using: (1) cassandra-env.sh (2) cassandra-rackdc.properties (3) cassandra.yaml (4) jvm.options

All the other configuration files are pointed back to the conf/ directory from the tarball using an environment variable.

(2) and (3) set the properties of the node, such as its name, turning on gossip, etc:

(1) and (4) are used by the cassandra bash script when it launches -- the script goes looking for additional configuration files. We've set it so that the cassandra.in.sh is found in the install area; but previously (1) and (4) were not being found, and these are required to activate JMX, port 7199, which nodetool uses to connect to the cluster.