Example Script¶
This example command-line script file is part of the es_client source code and is at
./es_client/cli_example.py
. The wrapper script run_script.py
is at the root-level of the
code at ./run_script.py
and will automatically target the cli_example.py
script.
es_client
in Action¶
Whether you have a running version of Elasticsearch or not, you can execute this script as outlined so long as the Python dependencies are installed. If you’ve cloned the github repository, this can be done by running the following command:
Install Prerequisites¶
Warning
I highly recommend setting up a Python virtualenv of some kind before running pip
pip install -U '.[doc,test]'
Run the Script with --help
or -h
¶
With the dependencies installed, the script should just run:
python run_script.py --help
Running the command will show the command-line help/usage output:
Output¶
Usage: run_script.py [OPTIONS] COMMAND [ARGS]...
CLI Example
Any text added to a docstring will show up in the --help/usage output.
Set short_help='' in @func.command() definitions for each command for terse descriptions in the main help/usage output, as
with show_all_options() in this example.
Options:
--config PATH Path to configuration file.
--hosts TEXT Elasticsearch URL to connect to.
--cloud_id TEXT Elastic Cloud instance id
--api_token TEXT The base64 encoded API Key token
--id TEXT API Key "id" value
--api_key TEXT API Key "api_key" value
--username TEXT Elasticsearch username
--password TEXT Elasticsearch password
--request_timeout FLOAT Request timeout in seconds
--verify_certs / --no-verify_certs
Verify SSL/TLS certificate(s) [default: verify_certs]
--ca_certs TEXT Path to CA certificate file or directory
--client_cert TEXT Path to client certificate file
--client_key TEXT Path to client key file
--loglevel [DEBUG|INFO|WARNING|ERROR|CRITICAL]
Log level
--logfile TEXT Log file
--logformat [default|json|ecs] Log output format
-v, --version Show the version and exit.
-h, --help Show this message and exit.
Commands:
show-all-options Show all configuration options
test-connection Test connection to Elasticsearch
Run the Script with a Command¶
At the bottom of the usage/help output, you should see show-all-options
and test-connection
.
Let’s re-run the script with show-all-options
:
python run_script.py show-all-options
Perhaps you’re confused to see another help/usage output. But there’s a difference:
Output¶
Usage: run_script.py show-all-options [OPTIONS]
ALL OPTIONS SHOWN
The full list of options available for configuring a connection at the command-line.
Options:
--config PATH Path to configuration file. [env var: ESCLIENT_CONFIG]
--hosts TEXT Elasticsearch URL to connect to. [env var: ESCLIENT_HOSTS]
--cloud_id TEXT Elastic Cloud instance id [env var: ESCLIENT_CLOUD_ID]
--api_token TEXT The base64 encoded API Key token [env var: ESCLIENT_API_TOKEN]
--id TEXT API Key "id" value [env var: ESCLIENT_ID]
--api_key TEXT API Key "api_key" value [env var: ESCLIENT_API_KEY]
--username TEXT Elasticsearch username [env var: ESCLIENT_USERNAME]
--password TEXT Elasticsearch password [env var: ESCLIENT_PASSWORD]
--bearer_auth TEXT Bearer authentication token [env var: ESCLIENT_BEARER_AUTH]
--opaque_id TEXT X-Opaque-Id HTTP header value [env var: ESCLIENT_OPAQUE_ID]
--request_timeout FLOAT Request timeout in seconds [env var: ESCLIENT_REQUEST_TIMEOUT]
--http_compress / --no-http_compress
Enable HTTP compression [env var: ESCLIENT_HTTP_COMPRESS]
--verify_certs / --no-verify_certs
Verify SSL/TLS certificate(s) [env var: ESCLIENT_VERIFY_CERTS]
--ca_certs TEXT Path to CA certificate file or directory [env var: ESCLIENT_CA_CERTS]
--client_cert TEXT Path to client certificate file [env var: ESCLIENT_CLIENT_CERT]
--client_key TEXT Path to client key file [env var: ESCLIENT_CLIENT_KEY]
--ssl_assert_hostname TEXT Hostname or IP address to verify on the node's certificate. [env var:
ESCLIENT_SSL_ASSERT_HOSTNAME]
--ssl_assert_fingerprint TEXT SHA-256 fingerprint of the node's certificate. If this value is given then root-of-trust
verification isn't done and only the node's certificate fingerprint is verified. [env var:
ESCLIENT_SSL_ASSERT_FINGERPRINT]
--ssl_version TEXT Minimum acceptable TLS/SSL version [env var: ESCLIENT_SSL_VERSION]
--master-only / --no-master-only
Only run if the single host provided is the elected master [env var: ESCLIENT_MASTER_ONLY]
--skip_version_test / --no-skip_version_test
Elasticsearch version compatibility check [env var: ESCLIENT_SKIP_VERSION_TEST]
--loglevel [DEBUG|INFO|WARNING|ERROR|CRITICAL]
Log level [env var: ESCLIENT_LOGLEVEL]
--logfile TEXT Log file [env var: ESCLIENT_LOGFILE]
--logformat [default|json|ecs] Log output format [env var: ESCLIENT_LOGFORMAT]
--blacklist TEXT Named entities will not be logged [env var: ESCLIENT_BLACKLIST]
-v, --version Show the version and exit.
-h, --help Show this message and exit.
Run the Script with a Command (continued)¶
A closer look will show that this help output is slightly different, and shows options that the first run did not. This is on purpose. This is to show how you can use Click to show or hide options at the command line. This can be done for multiple reasons, including hiding sensitive information. In this case, however, it’s mostly to keep things clean and as terse as possible by showing only the most frequently used options.
Run the Script with a live host¶
Now that we’ve come this far, it’s time to run against a live instance of Elasticsearch!
Let’s re-run the script with the command test-connection
. This time, unless we’re using a local
instance of Elasticsearch running on the default URL of http://127.0.0.1:9200, we will need to
specify a few options. Your options may vary, but let’s assume you have an Elasticsearch instance in
Elastic Cloud and you have a cloud_id and an API key to use:
If my cloud_id were example:REDACTED
, and my API key was also apikey:REDACTED
, I could run:
python run_script.py --cloud_id example:REDACTED --api_token apikey:REDACTED test-connection
If your API key came in two pieces rather than the base64 encoded single token, that’s okay! You can make that work, too:
python run_script.py --cloud_id example:REDACTED --api_key KEYVALUE --id IDVALUE test-connection
Or maybe you don’t have a cloud_id, but you have a URL, and a username and a password:
python run_script.py --hosts URL --username USER --password PASS test-connection
Maybe you have a YAML configuration file with all the options you need to use:
python run_script.py --config /path/to/config.yaml test-connection
There are so many ways you can slice and dice this!
Output¶
If all went well, you should see something like this:
Connection result:
{'name': 'NODENAME', 'cluster_name': 'CLUSTERNAME', 'cluster_uuid': 'UUID', 'version':
{'number': '8.12.0', 'build_flavor': 'default', 'build_type': 'docker', 'build_hash':
'HASH', 'build_date': '2024-01-11T10:05:27.953830042Z', 'build_snapshot': False,
'lucene_version': '9.9.1', 'minimum_wire_compatibility_version': '7.17.0',
'minimum_index_compatibility_version': '7.0.0'}, 'tagline': 'You Know, for Search'}
Option Errata¶
Most of the options should be straightforward, but a few should be explained.
Multiples¶
The command-line options --hosts
and --blacklist
can be used multiple times in the same
command-line, e.g.
python run_script.py --hosts http://127.0.0.1:9200 --hosts http://127.0.0.2:9200 ...
This is especially nice for reducing log volume with log blacklisting! See one you don’t want or
need? Run it again with another --blacklist
entry!
Configuration File Override¶
You can use a YAML configuration file for all options. But you can also mix configuration file settings with command-line options. The thing to know is that command-line options will always supersede settings in a configuration file.
ENVIRONMENT VARIABLES¶
Click makes it easy to use environment variables to pass values to options. In fact, it’s now built
in to es_client
! Any option can have an environment variable. All you need to do is prefix the
uppercase name of the option with ESCLIENT_
and replace any hyphens in the option name with
underscores. You may have noticed that the environment variables were shown in the
show-all-options
output above and wondered what that meant. Well, now you know!
ESCLIENT_LOGLEVEL=DEBUG python run_script.py --hosts http://127.0.0.1:9200
Congratulations, you’ve now set loglevel to DEBUG with an environment variable!
Multiples?¶
How do environment variables work for parameters that can have multiple values?
Great question! For the options es_client
has that can do multiples, namely hosts
and
blacklist
, you need to put all values into a single environment variable and separate them with
whitespace:
ESCLIENT_HOSTS='http://127.0.0.1:9200 http://localhost:9200' python run_script.py test-connection
A quick look at the DEBUG log shows the following (redacted for brevity):
... "Elasticsearch Configuration" config: {'hosts': ['http://127.0.0.1:9200', 'http://localhost:9200'], ...
Yup! Multiple values from a single environment variable is possible!
Flags, or boolean options?¶
A quick look at the show-all-options
output reveals that our boolean options (i.e., those with
an on and off switch) show the defaults as the flag and not as True or False:
--http_compress / --no-http_compress
Enable HTTP compression [env var: ESCLIENT_HTTP_COMPRESS; default: no-http_compress]
Does this mean you have to set ES_CLIENT_COMPRESS
to http_compress
or no-http_compress
?
No. In fact, don’t do that. Click is very smart and can interpret most boolean-esque settings.
True values: 1, True, true, TRUE (pretty sure it’s case-insensitive) False values: 0, False, false, FALSE
So here’s the real-world example:
ESCLIENT_HTTP_COMPRESS=1 python run_script.py test-connection
And in the debug log output (redacted for brevity):
"Elasticsearch Configuration" config: {'client': {'hosts': ..., 'http_compress': True,
You can take my word for it, or you can test it for yourself. It works.
Next Step: Make Your Own App Using es_client
¶
Visit the Tutorial for the next step!
File Source Code¶
This file is part of the source code and is at ./es_client/cli_example.py
.
"""Sample CLI script that will get a client using both config file and CLI args/options"""
# pylint: disable=unused-import, no-value-for-parameter, invalid-name, redefined-builtin
import click
from elasticsearch8.exceptions import BadRequestError, NotFoundError
from es_client.helpers import config as cfg
from es_client.defaults import OPTION_DEFAULTS, SHOW_EVERYTHING
from es_client.helpers.logging import configure_logging
# Since this file will not be imported, we use this to squash the F401 error
__all__ = ['BadRequestError', 'NotFoundError']
# The following default options are all automatically added by the decorator:
#
# ``@cfg.options_from_dict(OPTION_DEFAULTS)``
#
# Be sure to add any other options or arguments either before or after this decorator, and add any
# added arguments in ``def run()``, preserving their order in both locations. ``ctx`` needs to be
# the first arg after ``def run()`` as a special argument for Click, and does not need a decorator
# function.
# These options require the following other includes:
#
# from es_client.defaults import LOGGING_SETTINGS, ONOFF
# from es_client.helpers.utils import option_wrapper
# click_opt_wrap = option_wrapper()
#
# @click_opt_wrap(*cli_opts('config'))
# @click_opt_wrap(*cli_opts('hosts'))
# @click_opt_wrap(*cli_opts('cloud_id'))
# @click_opt_wrap(*cli_opts('api_token'))
# @click_opt_wrap(*cli_opts('id'))
# @click_opt_wrap(*cli_opts('api_key'))
# @click_opt_wrap(*cli_opts('username'))
# @click_opt_wrap(*cli_opts('password'))
# @click_opt_wrap(*cli_opts('bearer_auth'))
# @click_opt_wrap(*cli_opts('opaque_id'))
# @click_opt_wrap(*cli_opts('request_timeout'))
# @click_opt_wrap(*cli_opts('http_compress', onoff=ONOFF))
# @click_opt_wrap(*cli_opts('verify_certs', onoff=ONOFF))
# @click_opt_wrap(*cli_opts('ca_certs'))
# @click_opt_wrap(*cli_opts('client_cert'))
# @click_opt_wrap(*cli_opts('client_key'))
# @click_opt_wrap(*cli_opts('ssl_assert_hostname'))
# @click_opt_wrap(*cli_opts('ssl_assert_fingerprint'))
# @click_opt_wrap(*cli_opts('ssl_version'))
# @click_opt_wrap(*cli_opts('master-only', onoff=ONOFF))
# @click_opt_wrap(*cli_opts('skip_version_test', onoff=ONOFF))
# @click_opt_wrap(*cli_opts('loglevel', settings=LOGGING_SETTINGS))
# @click_opt_wrap(*cli_opts('logfile', settings=LOGGING_SETTINGS))
# @click_opt_wrap(*cli_opts('logformat', settings=LOGGING_SETTINGS))
# @click_opt_wrap(*cli_opts('blacklist', settings=LOGGING_SETTINGS))
# pylint: disable=unused-argument, redefined-builtin, too-many-arguments, too-many-locals, line-too-long
@click.group(context_settings=cfg.context_settings())
@cfg.options_from_dict(OPTION_DEFAULTS)
@click.version_option(None, '-v', '--version', prog_name="cli_example")
@click.pass_context
def run(
ctx, config, hosts, cloud_id, api_token, id, api_key, username, password, bearer_auth,
opaque_id, request_timeout, http_compress, verify_certs, ca_certs, client_cert, client_key,
ssl_assert_hostname, ssl_assert_fingerprint, ssl_version, master_only, skip_version_test,
loglevel, logfile, logformat, blacklist):
"""
CLI Example
Any text added to a docstring will show up in the --help/usage output.
Set short_help='' in @func.command() definitions for each command for terse descriptions in the
main help/usage output, as with show_all_options() in this example.
"""
# If there's a default file location for client configuration, e.g. $HOME/.curator/curator.yml,
# then specify it here. ctx.obj is now instantiated in ``helpers.config.cfg.context_settings()``
ctx.obj['default_config'] = None
# The ``cfg.get_config`` function will grab the configuration derived from a YAML config file
# specified in command-line parameters, or if that is unspecified but
# ctx.obj['default_config'] is provided, use that. If quiet=True, suppress the line written to
# STDOUT that indicates the file at ctx.obj['default_config'] is being used.
# If neither ctx.params['config'] nor ctx.obj['default_config'] reference a YAML configuration
# file, then a config dict with empty/default configured is generated.
# The result is stored in ctx.obj['draftcfg']
cfg.get_config(ctx, quiet=False)
# Configure logging. This will use the values from command line parameters, or what's now been
# stored in ctx.obj['draftcfg']
configure_logging(ctx)
# The ``cfg.generate_configdict`` function does all of the overriding of YAML config file options
# by command-line specified ones and stores the ready-to-be-used by Builder configuration in
# ctx.obj['configdict']
cfg.generate_configdict(ctx)
# SHOW ALL OPTIONS
#
# Below is the ``show-all-options`` command which overrides the default with the values in the
# OVERRIDE constant (``hidden: False`` and ``show_env_vars: True``) which will reveal any hidden by
# default options in the top-level menu so they are exposed in the --help output, as well as show
# the environment variable name that can be used to set the option without a flag/argument.
# The below options are all included automatically by the decorator:
#
# ``@cfg.options_from_dict(SHOW_EVERYTHING)``
#
# These options require the following other includes:
#
# from es_client.defaults import LOGGING_SETTINGS, ONOFF, OVERRIDE
# from es_client.helpers.utils import option_wrapper
# click_opt_wrap = option_wrapper()
#
# @click_opt_wrap(*cli_opts('config', override=OVERRIDE))
# @click_opt_wrap(*cli_opts('hosts', override=OVERRIDE))
# @click_opt_wrap(*cli_opts('cloud_id', override=OVERRIDE))
# @click_opt_wrap(*cli_opts('api_token', override=OVERRIDE))
# @click_opt_wrap(*cli_opts('id', override=OVERRIDE))
# @click_opt_wrap(*cli_opts('api_key', override=OVERRIDE))
# @click_opt_wrap(*cli_opts('username', override=OVERRIDE))
# @click_opt_wrap(*cli_opts('password', override=OVERRIDE))
# @click_opt_wrap(*cli_opts('bearer_auth', override=OVERRIDE))
# @click_opt_wrap(*cli_opts('opaque_id', override=OVERRIDE))
# @click_opt_wrap(*cli_opts('request_timeout', override=OVERRIDE))
# @click_opt_wrap(*cli_opts('http_compress', onoff=ONOFF, override=OVERRIDE))
# @click_opt_wrap(*cli_opts('verify_certs', onoff=ONOFF, override=OVERRIDE))
# @click_opt_wrap(*cli_opts('ca_certs', override=OVERRIDE))
# @click_opt_wrap(*cli_opts('client_cert', override=OVERRIDE))
# @click_opt_wrap(*cli_opts('client_key', override=OVERRIDE))
# @click_opt_wrap(*cli_opts('ssl_assert_hostname', override=OVERRIDE))
# @click_opt_wrap(*cli_opts('ssl_assert_fingerprint', override=OVERRIDE))
# @click_opt_wrap(*cli_opts('ssl_version', override=OVERRIDE))
# @click_opt_wrap(*cli_opts('master-only', onoff=ONOFF, override=OVERRIDE))
# @click_opt_wrap(*cli_opts('skip_version_test', onoff=ONOFF, override=OVERRIDE))
# @click_opt_wrap(*cli_opts('loglevel', settings=LOGGING_SETTINGS, override=OVERRIDE))
# @click_opt_wrap(*cli_opts('logfile', settings=LOGGING_SETTINGS, override=OVERRIDE))
# @click_opt_wrap(*cli_opts('logformat', settings=LOGGING_SETTINGS, override=OVERRIDE))
# @click_opt_wrap(*cli_opts('blacklist', settings=LOGGING_SETTINGS, override=OVERRIDE))
# NOTE: Different procedure for show_all_options than other sub-commands
# Normally, for a sub-command, you would not reset the `cfg.context_settings` as we've done here
# because it also resets the context (ctx). We normally want to pass this along from the top level
# command. In this case, we want it to look like the root-level command for the sake of the
# environment variables being shown for the root-level and not a sub-level command.
# pylint: disable=unused-argument, redefined-builtin, too-many-arguments, too-many-locals, line-too-long
@run.command(context_settings=cfg.context_settings(), short_help='Show all configuration options')
@cfg.options_from_dict(SHOW_EVERYTHING)
@click.version_option(None, '-v', '--version', prog_name="cli_example")
@click.pass_context
def show_all_options(
ctx, config, hosts, cloud_id, api_token, id, api_key, username, password, bearer_auth,
opaque_id, request_timeout, http_compress, verify_certs, ca_certs, client_cert, client_key,
ssl_assert_hostname, ssl_assert_fingerprint, ssl_version, master_only, skip_version_test,
loglevel, logfile, logformat, blacklist):
"""
ALL OPTIONS SHOWN
The full list of options available for configuring a connection at the command-line.
"""
ctx = click.get_current_context()
click.echo(ctx.get_help())
ctx.exit()
#
# Below is a way to run a command from the main command-line page. The Tutorial in the
# documentation shows how to take this example, copy it, and run your own code.
#
@run.command()
@click.pass_context
def test_connection(ctx):
"""
Test connection to Elasticsearch
"""
# Because of `@click.pass_context`, we can access `ctx.obj` here from the `run` function
# that made it:
client = cfg.get_client(configdict=ctx.obj['configdict'])
# If we're here, we'll see the output from GET http(s)://hostname.tld:PORT
click.secho('\nConnection result: ', bold=True)
click.secho(f'{client.info()}\n')
if __name__ == '__main__':
run()