Austin S. Hemmelgarn
Improve the system-info.sh script to report CPU and RAM meta-data. (#7815)
* Add CPU information collection for Linux and FreeBSD. This adds logic to system.info.sh to collect info about the system's CPU. It adds the following keys to the output of the script: * NETDATA_CPU_LOGICAL_CPU_COUNT: This reports the number of logical CPU cores the system is actually using (including offline ones). This may differ from the CPU's advertised core count, but is what most people actually care about. * NETDATA_CPU_VENDOR: This reports the CPU manufacturer. This is needed because some systems do not include the manufacturer name in the CPU model name. * NETDATA_CPU_MODEL: This reports the CPU model. It may or may not include any of a model number, intended operating frequency, and manufacturer name. * NETDATA_CPU_FREQ: This reports a best guess at the design frequency for the CPU. It may instead be the max boost frequency. This is reported as a number with associated units, which will usually be either hertz or megahertz. * NETDATA_CPU_DETECTION: This reports the method used to detect the CPU information, It will be either 'none' if no detection was successful, or a space-separated list of detection methods. This may potentially use any of the following detection methods: * lscpu: Uses a mix of information from across the system. Requires the `lscpu` command to be installed. * dmidecode: Uses the information from the DMI tables. Requires hardware support as well as the `dmidecode` command. * nproc: Uses the `nproc` command from the GNU coreutils to get a count of logical processors. * sysctl: Uses the `sysctl` command on FreeBSD to fetch information. * sysfs: Uses /sys on Linux to fetch information. * procfs: Uses /proc/cpuinfo on Linux to fetch information. * uname: Uses the `uname` command from the GNU coreutils to get CPU model and vendor information. All values tht were not successfully detected should read back as 'unknown'. Some values may have spaces present, and thus are quoted in the output. * Collect total system RAM info in system-info.sh This collects info about the total usable system RAM in the system-info.sh script. It adds the following two keys to the output of the script: * NETDATA_TOTAL_RAM: Reports the total usable system RAM as a number with an associated unit, usually as bytes or kilobytes. Reports 'unknown' if this couldn't be determined. * NETDATA_RAM_DETECTION: Indicates how we detected the total RAM, or 'none' if we couldn't figure out the total RAM. * Make lscpu output parsing more robust. * Remove extra quotes. The output is not parsed as shell variables, but using a special parser that just reads everything from the `=` to EOL as the value. * Coerce output to base units. This properly converts the output for CPU frequencies and RAM sizes to use base units of Hertz or bytes, allowing for simpler parsing of the output. * Fix incorrect number handling in total RAM parsing. * Correctly fix incorrect number handling in total RAM parsing. * Fix parsing of `lscpu` output. This properly recognizes the CPU frequency value as MHz and truncates the value to an integer.
|2 hours ago|
|.github||1 day ago|
|.travis||1 week ago|
|backends||2 weeks ago|
|build||9 months ago|
|claim||1 month ago|
|cli||1 month ago|
|collectors||8 hours ago|
|contrib||14 hours ago|
|daemon||2 hours ago|
|database||3 days ago|
|diagrams||2 months ago|
|docs||21 hours ago|
|exporting||1 day ago|
|health||1 week ago|
|libnetdata||2 weeks ago|
|packaging||3 hours ago|
|registry||2 months ago|
|streaming||3 days ago|
|system||2 months ago|
|tests||6 days ago|
|web||3 days ago|
|.clang-format||4 months ago|
|.codacy.yml||11 months ago|
|.codeclimate.yml||1 year ago|
|.csslintrc||3 years ago|
|.eslintignore||3 years ago|
|.eslintrc||3 years ago|
|.gitattributes||6 months ago|
|.gitignore||1 month ago|
|.gitlab-ci.yml||4 months ago|
|.lgtm.yml||1 year ago|
|.remarkignore||5 months ago|
|.remarkrc.js||1 month ago|
|.squash.yml||4 months ago|
|.travis.yml||1 day ago|
|BUILD.md||1 week ago|
|CHANGELOG.md||16 hours ago|
|CMakeLists.txt||1 day ago|
|CODE_OF_CONDUCT.md||5 months ago|
|CONTRIBUTING.md||1 week ago|
|CONTRIBUTORS.md||3 months ago|
|DOCUMENTATION.md||6 days ago|
|Dockerfile.docs||1 week ago|
|HISTORICAL_CHANGELOG.md||1 week ago|
|LICENSE||1 year ago|
|Makefile.am||1 day ago|
|README.md||1 week ago|
|REDISTRIBUTED.md||5 months ago|
|SECURITY.md||3 months ago|
|configs.signatures||3 months ago|
|configure.ac||1 week ago|
|coverity-scan.sh||4 months ago|
|cppcheck.sh||1 year ago|
|netdata-installer.sh||4 hours ago|
|netdata.cppcheck||1 year ago|
|netdata.spec.in||4 hours ago|
|netlify.toml||1 month ago|
|package-lock.json||3 weeks ago|
|package.json||4 months ago|
Netdata is distributed, real-time, performance and health monitoring for systems and applications. It is a highly-optimized monitoring agent you install on all your systems and containers.
Netdata provides unparalleled insights, in real-time, of everything happening on the systems it runs (including web servers, databases, applications), using highly interactive web dashboards. It can run autonomously, without any third-party components, or it can be integrated to existing monitoring toolchains (Prometheus, Graphite, OpenTSDB, Kafka, Grafana, and more).
Netdata is fast and efficient, designed to permanently run on all systems (physical & virtual servers, containers, IoT devices), without disrupting their core function.
Netdata is free, open-source software and it currently runs on Linux, FreeBSD, and MacOS, along with other systems derived from them, such as Kubernetes and Docker.
Netdata is not hosted by the CNCF but is the 3rd most starred open-source project in the Cloud Native Computing Foundation (CNCF) landscape.
People get addicted to Netdata. Once you use it on your systems, there is no going back! You've been warned…
The following animated GIF shows the top part of a typical Netdata dashboard.
A typical Netdata dashboard, in 1:1 timing. Charts can be panned by dragging them, zoomed in/out with
mouse wheel, an area can be selected for zoom-in with
mouse selection. Netdata is highly interactive, real-time, and optimized to get the work done!
Want to see Netdata live? Check out any of our live demos.
Netdata is used by hundreds of thousands of users all over the world. Check our GitHub watchers list. You will find people working for Amazon, Atos, Baidu, Cisco Systems, Citrix, Deutsche Telekom, DigitalOcean, Elastic, EPAM Systems, Ericsson, Google, Groupon, Hortonworks, HP, Huawei, IBM, Microsoft, NewRelic, Nvidia, Red Hat, SAP, Selectel, TicketMaster, Vimeo, and many more!
We provide Docker images for the most common architectures. These are statistics reported by Docker Hub:
When you install multiple Netdata, they are integrated into one distributed application, via a Netdata registry. This is a web browser feature and it allows us to count the number of unique users and unique Netdata servers installed. The following information comes from the global public Netdata registry we run:
To install Netdata from source on any Linux system (physical, virtual, container, IoT, edge) and keep it up to date with our nightly releases automatically, run the following:
# make sure you run `bash` for your shell bash # install Netdata directly from GitHub source bash <(curl -Ss https://my-netdata.io/kickstart.sh)
Starting with v1.12, Netdata collects anonymous usage information by default and sends it to Google Analytics. Read about the information collected, and learn how to-opt, on our anonymous statistics page.
The usage statistics are vital for us, as we use them to discover bugs and priortize new features. We thank you for actively contributing to Netdata's future.
To learn more about the pros and cons of using nightly vs. stable releases, see our notice about the two options.
The above command will:
More installation methods and additional options can be found at the installation page.
To try Netdata in a docker container, run this:
docker run -d --name=netdata \ -p 19999:19999 \ -v /etc/passwd:/host/etc/passwd:ro \ -v /etc/group:/host/etc/group:ro \ -v /proc:/host/proc:ro \ -v /sys:/host/sys:ro \ -v /var/run/docker.sock:/var/run/docker.sock:ro \ --cap-add SYS_PTRACE \ --security-opt apparmor=unconfined \ netdata/netdata
For more information about running Netdata with Docker, check the docker installation page.
From Netdata v1.12 and above, anonymous usage information is collected by default and sent to Google Analytics. To read more about the information collected and how to opt-out, check the anonymous statistics page.
Netdata has a quite different approach to monitoring.
Netdata is a monitoring agent you install on all your systems. It is:
All the above, are packaged together in a very flexible, extremely modular, distributed application.
This is how Netdata compares to other monitoring solutions:
|Netdata||others (open-source and commercial)|
|High resolution metrics (1s granularity)||Low resolution metrics (10s granularity at best)|
|Monitors everything, thousands of metrics per node||Monitor just a few metrics|
|UI is super fast, optimized for anomaly detection||UI is good for just an abstract view|
|Meaningful presentation, to help you understand the metrics||You have to know the metrics before you start|
|Install and get results immediately||Long preparation is required to get any useful results|
|Use it for troubleshooting performance problems||Use them to get statistics of past performance|
|Kills the console for tracing performance issues||The console is always required for troubleshooting|
|Requires zero dedicated resources||Require large dedicated resources|
Netdata is open-source, free, super fast, very easy, completely open, extremely efficient, flexible and integrate-able.
It has been designed by system administrators, DevOps engineers, and developers for to not just visualize metrics, but also troubleshoot complex performance problems.
Nov 27th, 2019 - Netdata v1.19.0 released!
Release v1.19.0 contains 2 new collectors, 19 bug fixes, 17 improvements, and 19 documentation updates.
We completed a major rewrite of our web log collector to dramatically improve its flexibility and performance. The new collector, written entirely in Go, can parse and chart logs from Nginx and Apache servers, and combines numerous improvements. Netdata now supports the LTSV log format, creates charts for TLS and cipher usage, and is amazingly fast. In a test using SSD storage, the collector parsed the logs for 200,000 requests in about 200ms, using 30% of a single core.
This Go-based collector also has powerful custom log parsing capabilities, which means we're one step closer to a generic application log parser for Netdata. We're continuing to work on this parser to support more application log formatting in the future.
We introduced more cmocka unit testing to Netdata. In this release, we're testing how Netdata's internal web server processes HTTP requests—the first step to improve the quality of code throughout, reduce bugs, and make refactoring easier. We wanted to validate the web server's behavior but needed to build a layer of parametric testing on top of the CMocka test runner. Read all about our process of testing and selecting cmocka on our blog post: Building an agile team's ‘safety harness’ with cmocka and FOSS.
Netdata's Unbound collector was also completely rewritten in Go to improve how it collects and displays metrics. This new version can get dozens of metrics, including details on queries, cache, uptime, and even show per-thread metrics. See our tutorial on enabling the new collector via Netdata's amazing auto-detection feature.
We fixed an error where invalid spikes appeared on certain charts by improving the incremental counter reset/wraparound detection algorithm.
Netdata is a highly efficient, highly modular, metrics management engine. Its lockless design makes it ideal for concurrent operations on the metrics.
This is how it works:
|Collect||Multiple independent data collection workers are collecting metrics from their sources using the optimal protocol for each application and push the metrics to the database. Each data collection worker has lockless write access to the metrics it collects.||
|Store||Metrics are first stored in RAM in a custom database engine that then “spills” historical metrics to disk for efficient long-term metrics storage.||
|Check||A lockless independent watchdog is evaluating health checks on the collected metrics, triggers alarms, maintains a health transaction log and dispatches alarm notifications.||
|Stream||A lockless independent worker is streaming metrics, in full detail and in real-time, to remote Netdata servers, as soon as they are collected.||
|Archive||A lockless independent worker is down-sampling the metrics and pushes them to backend time-series databases.||
|Query||Multiple independent workers are attached to the internal web server, servicing API requests, including data queries.||
The result is a highly efficient, low-latency system, supporting multiple readers and one writer on each metric.
This is a high level overview of Netdata feature set and architecture. Click it to to interact with it (it has direct links to our documentation).
This is what you should expect from Netdata:
To improve clarity on charts, Netdata dashboards present positive values for metrics representing
received and negative values for metrics representing
Netdata charts showing the bandwidth and packets of a network interface.
received is positive and
Netdata charts automatically zoom vertically, to visualize the variation of each metric within the visible time-frame.
stacked chart, automatically switches to an auto-scaled
area chart when a single dimension is
Charts on Netdata dashboards are synchronized to each other. There is no master chart. Any chart can be panned or zoomed at any time, and all other charts will follow.
Charts are panned by dragging them with the mouse. Charts can be zoomed in/out with
mouse wheel while the
mouse pointer is over a chart.
The visible time-frame (pan and zoom) is propagated from Netdata server to Netdata server when navigating via the My nodes menu.
To improve visual anomaly detection across charts, the user can highlight a time-frame (by pressing
mouse selection) on all charts.
A highlighted time-frame can be given by pressing
mouse selection on any chart. Netdata will highlight the
same range on all charts.
Highlighted ranges are propagated from Netdata server to Netdata server, when navigating via the My nodes menu.
Netdata data collection is extensible. You can monitor anything you can get a metric for. Our plugin API supports a variety of programming languages to make nearly anything a Netdata plugin: Go, Python, Node.js, Ruby, Java, Bash, Perl, and more!
tcclasses in real-time.
mod-status(v2.2, v2.4) and cache log statistics, for multiple servers.
stub-status, for multiple servers.
access.logfiles - extracting in real-time, web server and proxy performance metrics and applying several health checks, etc.
named.statsdump file to collect real-time performance metrics. All versions of bind after 9.6 are supported.
chronyccommand to collect chrony statistics (Frequency, Last offset, RMS offset, Residual freq, Root delay, Root dispersion, Skew, System time).
radclientcommand to provide freeradius statistics (authentication, accounting, proxy-authentication, proxy-accounting).
You can easily extend Netdata, by writing plugins that collect data from any source, using any computer language.
The Netdata documentation is at https://docs.netdata.cloud, but you can also find each page inside of Netdata's
repository itself in Markdown (
.md) files. You can find all our documentation by navigating the repository.
Here is a quick list of notable documents:
||Instructions to install Netdata on your systems.|
||Instructions to install Netdata using docker.|
||Information about the Netdata daemon and its configuration.|
||Information about data collection plugins.|
||How Netdata's health monitoring works, how to create your own alarms and how to configure alarm notification methods.|
||How to build hierarchies of Netdata servers, by streaming metrics between them.|
||Long term archiving of metrics to industry-standard time-series databases, like
||Learn how to query the Netdata API and the queries it supports.|
||Learn how to generate badges (SVG images) from live data.|
||Learn how to create custom Netdata dashboards.|
||Learn how to create Netdata dashboards on Atlassian's Confluence.|
You can also check all the other directories. Most of them have plenty of documentation.
We welcome contributions. Feel free to join the team!
To report bugs or get help, use GitHub's issues.
You can also find Netdata on:
Netdata is GPLv3+.
Netdata re-distributes other open-source tools and libraries. Please check the third party licenses.
When people first hear about a new product, they frequently ask if it is any good. A Hacker News user remarked:
Note to self: Starting immediately, all raganwald projects will have a “Is it any good?” section in the readme, and the answer shall be “yes.".
So, we follow the tradition…
These people seem to like it.