Example Script

This example command-line script file is part of the es_client source code and is at ./es_client/cli_example.py. The wrapper script run_script.py is at the root-level of the code at ./run_script.py and will automatically target the cli_example.py script.

es_client in Action

Whether you have a running version of Elasticsearch or not, you can execute this script as outlined so long as the Python dependencies are installed. If you’ve cloned the github repository, this can be done by running the following command:

Install Prerequisites

Warning

I highly recommend setting up a Python virtualenv of some kind before running pip

pip install -U '.[doc,test]'

Run the Script with --help or -h

With the dependencies installed, the script should just run:

python run_script.py --help

Running the command will show the command-line help/usage output:

Output

Usage: run_script.py [OPTIONS] COMMAND [ARGS]...

  CLI Example

  Any text added to a docstring will show up in the --help/usage output.

  Set short_help='' in @func.command() definitions for each command for terse descriptions in the main help/usage output, as
  with show_all_options() in this example.

Options:
  --config PATH                   Path to configuration file.
  --hosts TEXT                    Elasticsearch URL to connect to.
  --cloud_id TEXT                 Elastic Cloud instance id
  --api_token TEXT                The base64 encoded API Key token
  --id TEXT                       API Key "id" value
  --api_key TEXT                  API Key "api_key" value
  --username TEXT                 Elasticsearch username
  --password TEXT                 Elasticsearch password
  --request_timeout FLOAT         Request timeout in seconds
  --verify_certs / --no-verify_certs
                                  Verify SSL/TLS certificate(s)  [default: verify_certs]
  --ca_certs TEXT                 Path to CA certificate file or directory
  --client_cert TEXT              Path to client certificate file
  --client_key TEXT               Path to client key file
  --loglevel [DEBUG|INFO|WARNING|ERROR|CRITICAL]
                                  Log level
  --logfile TEXT                  Log file
  --logformat [default|json|ecs]  Log output format
  -v, --version                   Show the version and exit.
  -h, --help                      Show this message and exit.

Commands:
  show-all-options  Show all configuration options
  test-connection   Test connection to Elasticsearch

Run the Script with a Command

At the bottom of the usage/help output, you should see show-all-options and test-connection.

Let’s re-run the script with show-all-options:

python run_script.py show-all-options

Perhaps you’re confused to see another help/usage output. But there’s a difference:

Output

Usage: run_script.py show-all-options [OPTIONS]

  ALL OPTIONS SHOWN

  The full list of options available for configuring a connection at the command-line.

Options:
  --config PATH                   Path to configuration file.  [env var: ESCLIENT_CONFIG]
  --hosts TEXT                    Elasticsearch URL to connect to.  [env var: ESCLIENT_HOSTS]
  --cloud_id TEXT                 Elastic Cloud instance id  [env var: ESCLIENT_CLOUD_ID]
  --api_token TEXT                The base64 encoded API Key token  [env var: ESCLIENT_API_TOKEN]
  --id TEXT                       API Key "id" value  [env var: ESCLIENT_ID]
  --api_key TEXT                  API Key "api_key" value  [env var: ESCLIENT_API_KEY]
  --username TEXT                 Elasticsearch username  [env var: ESCLIENT_USERNAME]
  --password TEXT                 Elasticsearch password  [env var: ESCLIENT_PASSWORD]
  --bearer_auth TEXT              Bearer authentication token  [env var: ESCLIENT_BEARER_AUTH]
  --opaque_id TEXT                X-Opaque-Id HTTP header value  [env var: ESCLIENT_OPAQUE_ID]
  --request_timeout FLOAT         Request timeout in seconds  [env var: ESCLIENT_REQUEST_TIMEOUT]
  --http_compress / --no-http_compress
                                  Enable HTTP compression  [env var: ESCLIENT_HTTP_COMPRESS]
  --verify_certs / --no-verify_certs
                                  Verify SSL/TLS certificate(s)  [env var: ESCLIENT_VERIFY_CERTS]
  --ca_certs TEXT                 Path to CA certificate file or directory  [env var: ESCLIENT_CA_CERTS]
  --client_cert TEXT              Path to client certificate file  [env var: ESCLIENT_CLIENT_CERT]
  --client_key TEXT               Path to client key file  [env var: ESCLIENT_CLIENT_KEY]
  --ssl_assert_hostname TEXT      Hostname or IP address to verify on the node's certificate.  [env var:
                                  ESCLIENT_SSL_ASSERT_HOSTNAME]
  --ssl_assert_fingerprint TEXT   SHA-256 fingerprint of the node's certificate. If this value is given then root-of-trust
                                  verification isn't done and only the node's certificate fingerprint is verified.  [env var:
                                  ESCLIENT_SSL_ASSERT_FINGERPRINT]
  --ssl_version TEXT              Minimum acceptable TLS/SSL version  [env var: ESCLIENT_SSL_VERSION]
  --master-only / --no-master-only
                                  Only run if the single host provided is the elected master  [env var: ESCLIENT_MASTER_ONLY]
  --skip_version_test / --no-skip_version_test
                                  Elasticsearch version compatibility check  [env var: ESCLIENT_SKIP_VERSION_TEST]
  --loglevel [DEBUG|INFO|WARNING|ERROR|CRITICAL]
                                  Log level  [env var: ESCLIENT_LOGLEVEL]
  --logfile TEXT                  Log file  [env var: ESCLIENT_LOGFILE]
  --logformat [default|json|ecs]  Log output format  [env var: ESCLIENT_LOGFORMAT]
  --blacklist TEXT                Named entities will not be logged  [env var: ESCLIENT_BLACKLIST]
  -v, --version                   Show the version and exit.
  -h, --help                      Show this message and exit.

Run the Script with a Command (continued)

A closer look will show that this help output is slightly different, and shows options that the first run did not. This is on purpose. This is to show how you can use Click to show or hide options at the command line. This can be done for multiple reasons, including hiding sensitive information. In this case, however, it’s mostly to keep things clean and as terse as possible by showing only the most frequently used options.

Run the Script with a live host

Now that we’ve come this far, it’s time to run against a live instance of Elasticsearch!

Let’s re-run the script with the command test-connection. This time, unless we’re using a local instance of Elasticsearch running on the default URL of http://127.0.0.1:9200, we will need to specify a few options. Your options may vary, but let’s assume you have an Elasticsearch instance in Elastic Cloud and you have a cloud_id and an API key to use:

If my cloud_id were example:REDACTED, and my API key was also apikey:REDACTED, I could run:

python run_script.py --cloud_id example:REDACTED --api_token apikey:REDACTED test-connection

If your API key came in two pieces rather than the base64 encoded single token, that’s okay! You can make that work, too:

python run_script.py --cloud_id example:REDACTED --api_key KEYVALUE --id IDVALUE test-connection

Or maybe you don’t have a cloud_id, but you have a URL, and a username and a password:

python run_script.py --hosts URL --username USER --password PASS test-connection

Maybe you have a YAML configuration file with all the options you need to use:

python run_script.py --config /path/to/config.yaml test-connection

There are so many ways you can slice and dice this!

Output

If all went well, you should see something like this:

Connection result:
{'name': 'NODENAME', 'cluster_name': 'CLUSTERNAME', 'cluster_uuid': 'UUID', 'version':
{'number': '8.12.0', 'build_flavor': 'default', 'build_type': 'docker', 'build_hash':
'HASH', 'build_date': '2024-01-11T10:05:27.953830042Z', 'build_snapshot': False,
'lucene_version': '9.9.1', 'minimum_wire_compatibility_version': '7.17.0',
'minimum_index_compatibility_version': '7.0.0'}, 'tagline': 'You Know, for Search'}

Option Errata

Most of the options should be straightforward, but a few should be explained.

Multiples

The command-line options --hosts and --blacklist can be used multiple times in the same command-line, e.g.

python run_script.py --hosts http://127.0.0.1:9200 --hosts http://127.0.0.2:9200 ...

This is especially nice for reducing log volume with log blacklisting! See one you don’t want or need? Run it again with another --blacklist entry!

Configuration File Override

You can use a YAML configuration file for all options. But you can also mix configuration file settings with command-line options. The thing to know is that command-line options will always supersede settings in a configuration file.

ENVIRONMENT VARIABLES

Click makes it easy to use environment variables to pass values to options. In fact, it’s now built in to es_client! Any option can have an environment variable. All you need to do is prefix the uppercase name of the option with ESCLIENT_ and replace any hyphens in the option name with underscores. You may have noticed that the environment variables were shown in the show-all-options output above and wondered what that meant. Well, now you know!

ESCLIENT_LOGLEVEL=DEBUG python run_script.py --hosts http://127.0.0.1:9200

Congratulations, you’ve now set loglevel to DEBUG with an environment variable!

Multiples?

How do environment variables work for parameters that can have multiple values?

Great question! For the options es_client has that can do multiples, namely hosts and blacklist, you need to put all values into a single environment variable and separate them with whitespace:

ESCLIENT_HOSTS='http://127.0.0.1:9200 http://localhost:9200' python run_script.py test-connection

A quick look at the DEBUG log shows the following (redacted for brevity):

... "Elasticsearch Configuration" config: {'hosts': ['http://127.0.0.1:9200', 'http://localhost:9200'], ...

Yup! Multiple values from a single environment variable is possible!

Flags, or boolean options?

A quick look at the show-all-options output reveals that our boolean options (i.e., those with an on and off switch) show the defaults as the flag and not as True or False:

--http_compress / --no-http_compress
   Enable HTTP compression  [env var: ESCLIENT_HTTP_COMPRESS; default: no-http_compress]

Does this mean you have to set ES_CLIENT_COMPRESS to http_compress or no-http_compress?

No. In fact, don’t do that. Click is very smart and can interpret most boolean-esque settings.

True values: 1, True, true, TRUE (pretty sure it’s case-insensitive) False values: 0, False, false, FALSE

So here’s the real-world example:

ESCLIENT_HTTP_COMPRESS=1 python run_script.py test-connection

And in the debug log output (redacted for brevity):

"Elasticsearch Configuration" config: {'client': {'hosts': ..., 'http_compress': True,

You can take my word for it, or you can test it for yourself. It works.

Next Step: Make Your Own App Using es_client

Visit the Tutorial for the next step!

File Source Code

This file is part of the source code and is at ./es_client/cli_example.py.

"""Sample CLI script that will get a client using both config file and CLI args/options"""
# pylint: disable=unused-import, no-value-for-parameter, invalid-name, redefined-builtin
import click
from elasticsearch8.exceptions import BadRequestError, NotFoundError
from es_client.helpers import config as cfg
from es_client.defaults import OPTION_DEFAULTS, SHOW_EVERYTHING
from es_client.helpers.logging import configure_logging
# Since this file will not be imported, we use this to squash the F401 error
__all__ = ['BadRequestError', 'NotFoundError']

# The following default options are all automatically added by the decorator:
#
# ``@cfg.options_from_dict(OPTION_DEFAULTS)``
#
# Be sure to add any other options or arguments either before or after this decorator, and add any
# added arguments in ``def run()``, preserving their order in both locations.  ``ctx`` needs to be
# the first arg after ``def run()`` as a special argument for Click, and does not need a decorator
# function.

# These options require the following other includes:
#
# from es_client.defaults import LOGGING_SETTINGS, ONOFF
# from es_client.helpers.utils import option_wrapper
# click_opt_wrap = option_wrapper()
#
# @click_opt_wrap(*cli_opts('config'))
# @click_opt_wrap(*cli_opts('hosts'))
# @click_opt_wrap(*cli_opts('cloud_id'))
# @click_opt_wrap(*cli_opts('api_token'))
# @click_opt_wrap(*cli_opts('id'))
# @click_opt_wrap(*cli_opts('api_key'))
# @click_opt_wrap(*cli_opts('username'))
# @click_opt_wrap(*cli_opts('password'))
# @click_opt_wrap(*cli_opts('bearer_auth'))
# @click_opt_wrap(*cli_opts('opaque_id'))
# @click_opt_wrap(*cli_opts('request_timeout'))
# @click_opt_wrap(*cli_opts('http_compress', onoff=ONOFF))
# @click_opt_wrap(*cli_opts('verify_certs', onoff=ONOFF))
# @click_opt_wrap(*cli_opts('ca_certs'))
# @click_opt_wrap(*cli_opts('client_cert'))
# @click_opt_wrap(*cli_opts('client_key'))
# @click_opt_wrap(*cli_opts('ssl_assert_hostname'))
# @click_opt_wrap(*cli_opts('ssl_assert_fingerprint'))
# @click_opt_wrap(*cli_opts('ssl_version'))
# @click_opt_wrap(*cli_opts('master-only', onoff=ONOFF))
# @click_opt_wrap(*cli_opts('skip_version_test', onoff=ONOFF))
# @click_opt_wrap(*cli_opts('loglevel', settings=LOGGING_SETTINGS))
# @click_opt_wrap(*cli_opts('logfile', settings=LOGGING_SETTINGS))
# @click_opt_wrap(*cli_opts('logformat', settings=LOGGING_SETTINGS))
# @click_opt_wrap(*cli_opts('blacklist', settings=LOGGING_SETTINGS))


# pylint: disable=unused-argument, redefined-builtin, too-many-arguments, too-many-locals, line-too-long
@click.group(context_settings=cfg.context_settings())
@cfg.options_from_dict(OPTION_DEFAULTS)
@click.version_option(None, '-v', '--version', prog_name="cli_example")
@click.pass_context
def run(
        ctx, config, hosts, cloud_id, api_token, id, api_key, username, password, bearer_auth,
        opaque_id, request_timeout, http_compress, verify_certs, ca_certs, client_cert, client_key,
        ssl_assert_hostname, ssl_assert_fingerprint, ssl_version, master_only, skip_version_test,
        loglevel, logfile, logformat, blacklist):
    """
    CLI Example

    Any text added to a docstring will show up in the --help/usage output.

    Set short_help='' in @func.command() definitions for each command for terse descriptions in the
    main help/usage output, as with show_all_options() in this example.
    """
    # If there's a default file location for client configuration, e.g. $HOME/.curator/curator.yml,
    # then specify it here. ctx.obj is now instantiated in ``helpers.config.cfg.context_settings()``
    ctx.obj['default_config'] = None

    # The ``cfg.get_config`` function will grab the configuration derived from a YAML config file
    # specified in command-line parameters, or if that is unspecified but
    # ctx.obj['default_config'] is provided, use that. If quiet=True, suppress the line written to
    # STDOUT that indicates the file at ctx.obj['default_config'] is being used.
    # If neither ctx.params['config'] nor ctx.obj['default_config'] reference a YAML configuration
    # file, then a config dict with empty/default configured is generated.
    # The result is stored in ctx.obj['draftcfg']
    cfg.get_config(ctx, quiet=False)

    # Configure logging. This will use the values from command line parameters, or what's now been
    # stored in ctx.obj['draftcfg']
    configure_logging(ctx)

    # The ``cfg.generate_configdict`` function does all of the overriding of YAML config file options
    # by command-line specified ones and stores the ready-to-be-used by Builder configuration in
    # ctx.obj['configdict']
    cfg.generate_configdict(ctx)

# SHOW ALL OPTIONS
#
# Below is the ``show-all-options`` command which overrides the default with the values in the
# OVERRIDE constant (``hidden: False`` and ``show_env_vars: True``) which will reveal any hidden by
# default options in the top-level menu so they are exposed in the --help output, as well as show
# the environment variable name that can be used to set the option without a flag/argument.

# The below options are all included automatically by the decorator:
#
# ``@cfg.options_from_dict(SHOW_EVERYTHING)``
#
# These options require the following other includes:
#
# from es_client.defaults import LOGGING_SETTINGS, ONOFF, OVERRIDE
# from es_client.helpers.utils import option_wrapper
# click_opt_wrap = option_wrapper()
#
# @click_opt_wrap(*cli_opts('config', override=OVERRIDE))
# @click_opt_wrap(*cli_opts('hosts', override=OVERRIDE))
# @click_opt_wrap(*cli_opts('cloud_id', override=OVERRIDE))
# @click_opt_wrap(*cli_opts('api_token', override=OVERRIDE))
# @click_opt_wrap(*cli_opts('id', override=OVERRIDE))
# @click_opt_wrap(*cli_opts('api_key', override=OVERRIDE))
# @click_opt_wrap(*cli_opts('username', override=OVERRIDE))
# @click_opt_wrap(*cli_opts('password', override=OVERRIDE))
# @click_opt_wrap(*cli_opts('bearer_auth', override=OVERRIDE))
# @click_opt_wrap(*cli_opts('opaque_id', override=OVERRIDE))
# @click_opt_wrap(*cli_opts('request_timeout', override=OVERRIDE))
# @click_opt_wrap(*cli_opts('http_compress', onoff=ONOFF, override=OVERRIDE))
# @click_opt_wrap(*cli_opts('verify_certs', onoff=ONOFF, override=OVERRIDE))
# @click_opt_wrap(*cli_opts('ca_certs', override=OVERRIDE))
# @click_opt_wrap(*cli_opts('client_cert', override=OVERRIDE))
# @click_opt_wrap(*cli_opts('client_key', override=OVERRIDE))
# @click_opt_wrap(*cli_opts('ssl_assert_hostname', override=OVERRIDE))
# @click_opt_wrap(*cli_opts('ssl_assert_fingerprint', override=OVERRIDE))
# @click_opt_wrap(*cli_opts('ssl_version', override=OVERRIDE))
# @click_opt_wrap(*cli_opts('master-only', onoff=ONOFF, override=OVERRIDE))
# @click_opt_wrap(*cli_opts('skip_version_test', onoff=ONOFF, override=OVERRIDE))
# @click_opt_wrap(*cli_opts('loglevel', settings=LOGGING_SETTINGS, override=OVERRIDE))
# @click_opt_wrap(*cli_opts('logfile', settings=LOGGING_SETTINGS, override=OVERRIDE))
# @click_opt_wrap(*cli_opts('logformat', settings=LOGGING_SETTINGS, override=OVERRIDE))
# @click_opt_wrap(*cli_opts('blacklist', settings=LOGGING_SETTINGS, override=OVERRIDE))

# NOTE: Different procedure for show_all_options than other sub-commands
# Normally, for a sub-command, you would not reset the `cfg.context_settings` as we've done here
# because it also resets the context (ctx). We normally want to pass this along from the top level
# command. In this case, we want it to look like the root-level command for the sake of the
# environment variables being shown for the root-level and not a sub-level command.


# pylint: disable=unused-argument, redefined-builtin, too-many-arguments, too-many-locals, line-too-long
@run.command(context_settings=cfg.context_settings(), short_help='Show all configuration options')
@cfg.options_from_dict(SHOW_EVERYTHING)
@click.version_option(None, '-v', '--version', prog_name="cli_example")
@click.pass_context
def show_all_options(
        ctx, config, hosts, cloud_id, api_token, id, api_key, username, password, bearer_auth,
        opaque_id, request_timeout, http_compress, verify_certs, ca_certs, client_cert, client_key,
        ssl_assert_hostname, ssl_assert_fingerprint, ssl_version, master_only, skip_version_test,
        loglevel, logfile, logformat, blacklist):
    """
    ALL OPTIONS SHOWN

    The full list of options available for configuring a connection at the command-line.
    """
    ctx = click.get_current_context()
    click.echo(ctx.get_help())
    ctx.exit()

#
# Below is a way to run a command from the main command-line page. The Tutorial in the
# documentation shows how to take this example, copy it, and run your own code.
#


@run.command()
@click.pass_context
def test_connection(ctx):
    """
    Test connection to Elasticsearch
    """
    # Because of `@click.pass_context`, we can access `ctx.obj` here from the `run` function
    # that made it:
    client = cfg.get_client(configdict=ctx.obj['configdict'])

    # If we're here, we'll see the output from GET http(s)://hostname.tld:PORT
    click.secho('\nConnection result: ', bold=True)
    click.secho(f'{client.info()}\n')


if __name__ == '__main__':
    run()