Go to file
Costa Tsaousis 5e1b95cf92
Deduplicate all netdata strings (#13570)
* rrdfamily

* rrddim

* rrdset plugin and module names

* rrdset units

* rrdset type

* rrdset family

* rrdset title

* rrdset title more

* rrdset context

* rrdcalctemplate context and removal of context hash from rrdset

* strings statistics

* rrdset name

* rearranged members of rrdset

* eliminate rrdset name hash; rrdcalc chart converted to STRING

* rrdset id, eliminated rrdset hash

* rrdcalc, alarm_entry, alert_config and some of rrdcalctemplate

* rrdcalctemplate

* rrdvar

* eval_variable

* rrddimvar and rrdsetvar

* rrdhost hostname, os and tags

* fix master commits

* added thread cache; implemented string_dup without locks

* faster thread cache

* rrdset and rrddim now use dictionaries for indexing

* rrdhost now uses dictionary

* rrdfamily now uses DICTIONARY

* rrdvar using dictionary instead of AVL

* allocate the right size to rrdvar flag members

* rrdhost remaining char * members to STRING *

* better error handling on indexing

* strings now use a read/write lock to allow parallel searches to the index

* removed AVL support from dictionaries; implemented STRING with native Judy calls

* string releases should be negative

* only 31 bits are allowed for enum flags

* proper locking on strings

* string threading unittest and fixes

* fix lgtm finding

* fixed naming

* stream chart/dimension definitions at the beginning of a streaming session

* thread stack variable is undefined on thread cancel

* rrdcontext garbage collect per host on startup

* worker control in garbage collection

* relaxed deletion of rrdmetrics

* type checking on dictfe

* netdata chart to monitor rrdcontext triggers

* Group chart label updates

* rrdcontext better handling of collected rrdsets

* rrdpush incremental transmition of definitions should use as much buffer as possible

* require 1MB per chart

* empty the sender buffer before enabling metrics streaming

* fill up to 50% of buffer

* reset signaling metrics sending

* use the shared variable for status

* use separate host flag for enabling streaming of metrics

* make sure the flag is clear

* add logging for streaming

* add logging for streaming on buffer overflow

* circular_buffer proper sizing

* removed obsolete logs

* do not execute worker jobs if not necessary

* better messages about compression disabling

* proper use of flags and updating rrdset last access time every time the obsoletion flag is flipped

* monitor stream sender used buffer ratio

* Update exporting unit tests

* no need to compare label value with strcmp

* streaming send workers now monitor bandwidth

* workers now use strings

* streaming receiver monitors incoming bandwidth

* parser shift of worker ids

* minor fixes

* Group chart label updates

* Populate context with dimensions that have data

* Fix chart id

* better shift of parser worker ids

* fix for streaming compression

* properly count received bytes

* ensure LZ4 compression ring buffer does not wrap prematurely

* do not stream empty charts; do not process empty instances in rrdcontext

* need_to_send_chart_definition() does not need an rrdset lock any more

* rrdcontext objects are collected, after data have been written to the db

* better logging of RRDCONTEXT transitions

* always set all variables needed by the worker utilization charts

* implemented double linked list for most objects; eliminated alarm indexes from rrdhost; and many more fixes

* lockless strings design - string_dup() and string_freez() are totally lockless when they dont need to touch Judy - only Judy is protected with a read/write lock

* STRING code re-organization for clarity

* thread_cache improvements; double numbers precision on worker threads

* STRING_ENTRY now shadown STRING, so no duplicate definition is required; string_length() renamed to string_strlen() to follow the paradigm of all other functions, STRING internal statistics are now only compiled with NETDATA_INTERNAL_CHECKS

* rrdhost index by hostname now cleans up; aclk queries of archieved hosts do not index hosts

* Add index to speed up database context searches

* Removed last_updated optimization (was also buggy after latest merge with master)

Co-authored-by: Stelios Fragkakis <52996999+stelfrag@users.noreply.github.com>
Co-authored-by: Vladimir Kobal <vlad@prokk.net>
2022-09-05 19:31:06 +03:00
.devcontainer Support VS Code container devenv (#10723) 2021-03-12 16:43:41 +02:00
.github Temporary fix for command injection vulnerability in GHA workflow. (#13600) 2022-08-31 14:23:59 -04:00
.travis Overhaul build CI. (#11699) 2021-11-15 10:02:57 -05:00
aclk Deduplicate all netdata strings (#13570) 2022-09-05 19:31:06 +03:00
build Make atomics a hard-dep. (#12730) 2022-05-02 17:59:40 +03:00
build_external feat: move dirs, logs, and env vars config options to separate sections (#12935) 2022-05-17 17:31:19 +03:00
claim Remove aclk_api.[ch] (#13540) 2022-08-24 10:41:14 +02:00
cli netdata doubles (#13217) 2022-06-28 17:04:37 +03:00
collectors Deduplicate all netdata strings (#13570) 2022-09-05 19:31:06 +03:00
contrib fix(packaging): add CAP_NET_ADMIN for go.d.plugin (#13507) 2022-08-11 18:57:35 +03:00
daemon Deduplicate all netdata strings (#13570) 2022-09-05 19:31:06 +03:00
database Deduplicate all netdata strings (#13570) 2022-09-05 19:31:06 +03:00
diagrams Docs: Removed Google Analytics tags (#12145) 2022-02-17 10:37:46 +00:00
docs Add link to the performance optimization guide (#13595) 2022-08-31 10:04:53 +03:00
exporting Deduplicate all netdata strings (#13570) 2022-09-05 19:31:06 +03:00
health Deduplicate all netdata strings (#13570) 2022-09-05 19:31:06 +03:00
libnetdata Deduplicate all netdata strings (#13570) 2022-09-05 19:31:06 +03:00
ml Deduplicate all netdata strings (#13570) 2022-09-05 19:31:06 +03:00
mqtt_websockets@0ccbce11b8 Fixes vbi parser in mqtt5 implementation (#13277) 2022-06-30 17:41:52 +02:00
packaging [ci skip] Update changelog and version for nightly build: v1.36.0-82-nightly. 2022-09-05 00:19:23 +00:00
parser Deduplicate all netdata strings (#13570) 2022-09-05 19:31:06 +03:00
registry Deduplicate all netdata strings (#13570) 2022-09-05 19:31:06 +03:00
spawn Trace rwlocks of netdata (#12785) 2022-05-03 00:31:50 +03:00
streaming Deduplicate all netdata strings (#13570) 2022-09-05 19:31:06 +03:00
system netdata.service: Update PIDFile to avoid systemd legacy path warning (#13504) 2022-08-15 08:51:12 -04:00
tests netdata doubles (#13217) 2022-06-28 17:04:37 +03:00
web Deduplicate all netdata strings (#13570) 2022-09-05 19:31:06 +03:00
.clang-format Fine tune clang-format (#7271) 2021-04-15 12:02:36 +03:00
.codacy.yml codacy/lgtm ignore judy sources (#13411) 2022-07-20 11:15:49 -04:00
.codeclimate.yml Remove node.d.plugin and relevant files (#12769) 2022-05-03 09:16:21 +03:00
.csslintrc added codeclimate coverage 2017-01-06 18:01:57 +02:00
.dockerignore Restore a broken symbolic link (#12923) 2022-05-16 18:46:36 +03:00
.eslintignore Bundle the react dashboard code into the agent repo directly. (#11139) 2021-05-14 11:41:16 -04:00
.eslintrc added codeclimate coverage 2017-01-06 18:01:57 +02:00
.gitattributes Add a .gitattributes file (#6381) 2019-07-05 11:54:32 +02:00
.gitignore include Judy into our source tree (#13362) 2022-07-22 16:55:06 +02:00
.gitmodules Anomaly Detection MVP (#11548) 2021-10-27 09:26:21 +03:00
.lgtm.yml codacy/lgtm ignore judy sources (#13411) 2022-07-20 11:15:49 -04:00
.mlc_config.json GitHub action markdown link check update (#10474) 2021-01-11 13:50:16 -05:00
.remarkignore add CHANGELOG.md to .remarkignore (#6671) 2019-08-15 16:41:08 -07:00
.remarkrc.js Change lint standard for lists (#10371) 2021-01-07 08:43:18 -07:00
.squash.yml Squash integration (#5566) 2019-09-16 16:49:31 +02:00
.travis.yml include Judy into our source tree (#13362) 2022-07-22 16:55:06 +02:00
.yamllint.yml Clean up YAML files in the repository. (#11570) 2021-09-27 10:16:39 -04:00
BREAKING_CHANGES.md Docs: Removed Google Analytics tags (#12145) 2022-02-17 10:37:46 +00:00
BUILD.md Docs: Removed Google Analytics tags (#12145) 2022-02-17 10:37:46 +00:00
CHANGELOG.md [ci skip] Update changelog and version for nightly build: v1.36.0-82-nightly. 2022-09-05 00:19:23 +00:00
CMakeLists.txt Deduplicate all netdata strings (#13570) 2022-09-05 19:31:06 +03:00
Dockerfile Remove the confusion around the multiple Dockerfile(s) we have (#8214) 2020-03-10 08:12:26 +10:00
Dockerfile.test Spelling build (#10909) 2021-04-14 12:24:45 +03:00
HISTORICAL_CHANGELOG.md Spelling md (#10508) 2021-01-18 07:43:43 -05:00
LICENSE remove license templates; add info about SPDX to main license file 2018-09-08 15:53:07 +02:00
Makefile.am Deduplicate all netdata strings (#13570) 2022-09-05 19:31:06 +03:00
README.md add discord, youtube, linkedin links to README (#13419) 2022-07-21 19:48:15 +01:00
REDISTRIBUTED.md Metric correlations (#12582) 2022-05-04 13:59:58 +03:00
build-artifacts.sh Fix the netdata-updater.sh to correctly pass REINSTALL_OPTIONS (finally) (#8808) 2020-08-25 15:25:26 +10:00
configs.signatures Drop dirty dbengine pages if disk cannot keep up (#7777) 2020-02-06 21:58:13 +02:00
configure.ac Trimmed-median, trimmed-mean and percentile (#13469) 2022-08-05 12:50:11 +03:00
coverity-scan.sh Bump Coverity version to latest (2022.06). (#13541) 2022-08-19 08:02:18 -04:00
cppcheck.sh optimized ses and added des (#4470) 2018-10-24 03:03:57 +03:00
netdata-installer.sh Remove extra U from log message (#13514) 2022-08-13 15:22:47 +03:00
netdata.cppcheck remove static dir config 2018-09-08 15:45:34 +02:00
netdata.spec.in fix(packaging): add CAP_NET_ADMIN for go.d.plugin (#13507) 2022-08-11 18:57:35 +03:00

README.md

Netdata

Netdata is high-fidelity infrastructure monitoring and troubleshooting.
Open-source, free, preconfigured, opinionated, and always real-time.


GitHub Stars
Latest release Nightly release
Build status CII Best Practices License: GPL v3+
Code Climate LGTM C LGTM PYTHON

---

Netdata's distributed, real-time monitoring Agent collects thousands of metrics from systems, hardware, containers, and applications with zero configuration. It runs permanently on all your physical/virtual servers, containers, cloud deployments, and edge/IoT devices, and is perfectly safe to install on your systems mid-incident without any preparation.

You can install Netdata on most Linux distributions (Ubuntu, Debian, CentOS, and more), container platforms (Kubernetes clusters, Docker), and many other operating systems (FreeBSD, macOS). No sudo required.

Netdata is designed by system administrators, DevOps engineers, and developers to collect everything, help you visualize metrics, troubleshoot complex performance problems, and make data interoperable with the rest of your monitoring stack.

People get addicted to Netdata. Once you use it on your systems, there's no going back! You've been warned...

Users who are addicted toNetdata

Menu

Features

Netdata inaction

Here's what you can expect from Netdata:

  • 1s granularity: The highest possible resolution for all metrics.
  • Unlimited metrics: Netdata collects all the available metrics—the more, the better.
  • 1% CPU utilization of a single core: It's unbelievably optimized.
  • A few MB of RAM: The highly-efficient database engine stores per-second metrics in RAM and then "spills" historical metrics to disk long-term storage.
  • Minimal disk I/O: While running, Netdata only writes historical metrics and reads error and access logs.
  • Zero configuration: Netdata auto-detects everything, and can collect up to 10,000 metrics per server out of the box.
  • Zero maintenance: You just run it. Netdata does the rest.
  • Stunningly fast, interactive visualizations: The dashboard responds to queries in less than 1ms per metric to synchronize charts as you pan through time, zoom in on anomalies, and more.
  • Visual anomaly detection: Our UI/UX emphasizes the relationships between charts to help you detect the root cause of anomalies.
  • Machine learning (ML) features out of the box: Unsupervised ML-based anomaly detection, every second, every metric, zero-config! Metric correlations to help with short-term change detection. And other additional ML-based features to help make your life easier.
  • Scales to infinity: You can install it on all your servers, containers, VMs, and IoT devices. Metrics are not centralized by default, so there is no limit.
  • Several operating modes: Autonomous host monitoring (the default), headless data collector, forwarding proxy, store and forward proxy, central multi-host monitoring, in all possible configurations. Use different metrics retention policies per node and run with or without health monitoring.

Netdata works with tons of applications, notifications platforms, and other time-series databases:

  • 300+ system, container, and application endpoints: Collectors autodetect metrics from default endpoints and immediately visualize them into meaningful charts designed for troubleshooting. See everything we support.
  • 20+ notification platforms: Netdata's health watchdog sends warning and critical alarms to your favorite platform to inform you of anomalies just seconds after they affect your node.
  • 30+ external time-series databases: Export resampled metrics as they're collected to other local- and Cloud-based databases for best-in-class interoperability.

💡 Want to leverage the monitoring power of Netdata across entire infrastructure? View metrics from any number of distributed nodes in a single interface and unlock even more features with Netdata Cloud.

Get Netdata

User base Servers monitored Sessions served Docker Hub pulls
New users today New machines today Sessions today Docker Hub pulls today

To install Netdata from source on most Linux systems (physical, virtual, container, IoT, edge), run our one-line installation script. This script downloads and builds all dependencies, including those required to connect to Netdata Cloud if you choose, and enables automatic nightly updates and anonymous statistics.

wget -O /tmp/netdata-kickstart.sh https://my-netdata.io/kickstart.sh && sh /tmp/netdata-kickstart.sh

To view the Netdata dashboard, navigate to http://localhost:19999, or http://NODE:19999.

Docker

You can also try out Netdata's capabilities in a Docker container:

docker run -d --name=netdata \
  -p 19999:19999 \
  -v netdataconfig:/etc/netdata \
  -v netdatalib:/var/lib/netdata \
  -v netdatacache:/var/cache/netdata \
  -v /etc/passwd:/host/etc/passwd:ro \
  -v /etc/group:/host/etc/group:ro \
  -v /proc:/host/proc:ro \
  -v /sys:/host/sys:ro \
  -v /etc/os-release:/host/etc/os-release:ro \
  --restart unless-stopped \
  --cap-add SYS_PTRACE \
  --security-opt apparmor=unconfined \
  netdata/netdata

To view the Netdata dashboard, navigate to http://localhost:19999, or http://NODE:19999.

Other operating systems

See our documentation for additional operating systems, including Kubernetes, .deb/.rpm packages, and more.

Post-installation

When you're finished with installation, check out our single-node or infrastructure monitoring quickstart guides based on your use case.

Or, skip straight to configuring the Netdata Agent.

Read through Netdata's documentation, which is structured based on actions and solutions, to enable features like health monitoring, alarm notifications, long-term metrics storage, exporting to external databases, and more.

Netdata Cloud

Netdata Cloud works with Netdata's free, open-source monitoring agent to help you monitor and troubleshoot every layer of your systems to find weaknesses before they turn into outages. Using both tools can help you turn data into insights immediately.

Get Netdata Cloud now!

How it works

Netdata is a highly efficient, highly modular, metrics management engine. Its lockless design makes it ideal for concurrent operations on the metrics.

Diagram of Netdata's corefunctionality

The result is a highly efficient, low-latency system, supporting multiple readers and one writer on each metric.

Infographic

This is a high-level overview of Netdata features and architecture. Click on it to view an interactive version, which has links to our documentation.

An infographic of how Netdataworks

Documentation

Netdata's documentation is available at Netdata Learn.

This site also hosts a number of guides to help newer users better understand how to collect metrics, troubleshoot via charts, export to external databases, and more.

Community

Netdata is an inclusive open-source project and community. Please read our Code of Conduct.

Find most of the Netdata team in our community forums. It's the best place to ask questions, find resources, and engage with passionate professionals. The team is also available and active in our Discord too.

You can also find Netdata on:

Contribute

Contributions are the lifeblood of open-source projects. While we continue to invest in and improve Netdata, we need help to democratize monitoring!

  • Read our Contributing Guide, which contains all the information you need to contribute to Netdata, such as improving our documentation, engaging in the community, and developing new features. We've made it as frictionless as possible, but if you need help, just ping us on our community forums!
  • We have a whole category dedicated to contributing and extending Netdata on our community forums
  • Found a bug? Open a GitHub issue.
  • View our Security Policy.

Package maintainers should read the guide on building Netdata from source for instructions on building each Netdata component from source and preparing a package.

License

The Netdata Agent is GPLv3+. Netdata re-distributes other open-source tools and libraries. Please check the third party licenses.

Is it any good?

Yes.

When people first hear about a new product, they frequently ask if it is any good. A Hacker News user remarked:

Note to self: Starting immediately, all raganwald projects will have a “Is it any good?” section in the readme, and the answer shall be “yes.".