Go to file
Bjørn Erik Pedersen eada236f87
Introduce a tree map for all content
This commit introduces a new data structure to store pages and their resources.

This data structure is backed by radix trees.

This simplies tree operations, makes all pages a bundle,  and paves the way for #6310.

It also solves a set of annoying issues (see list below).

Not a motivation behind this, but this commit also makes Hugo in general a little bit faster and more memory effective (see benchmarks). Especially for partial rebuilds on content edits, but also when taxonomies is in use.

```
name                                   old time/op    new time/op    delta
SiteNew/Bundle_with_image/Edit-16        1.32ms ± 8%    1.00ms ± 9%  -24.42%  (p=0.029 n=4+4)
SiteNew/Bundle_with_JSON_file/Edit-16    1.28ms ± 0%    0.94ms ± 0%  -26.26%  (p=0.029 n=4+4)
SiteNew/Tags_and_categories/Edit-16      33.9ms ± 2%    21.8ms ± 1%  -35.67%  (p=0.029 n=4+4)
SiteNew/Canonify_URLs/Edit-16            40.6ms ± 1%    37.7ms ± 3%   -7.20%  (p=0.029 n=4+4)
SiteNew/Deep_content_tree/Edit-16        56.7ms ± 0%    51.7ms ± 1%   -8.82%  (p=0.029 n=4+4)
SiteNew/Many_HTML_templates/Edit-16      19.9ms ± 2%    18.3ms ± 3%   -7.64%  (p=0.029 n=4+4)
SiteNew/Page_collections/Edit-16         37.9ms ± 4%    34.0ms ± 2%  -10.28%  (p=0.029 n=4+4)
SiteNew/Bundle_with_image-16             10.7ms ± 0%    10.6ms ± 0%   -1.15%  (p=0.029 n=4+4)
SiteNew/Bundle_with_JSON_file-16         10.8ms ± 0%    10.7ms ± 0%   -1.05%  (p=0.029 n=4+4)
SiteNew/Tags_and_categories-16           43.2ms ± 1%    39.6ms ± 1%   -8.35%  (p=0.029 n=4+4)
SiteNew/Canonify_URLs-16                 47.6ms ± 1%    47.3ms ± 0%     ~     (p=0.057 n=4+4)
SiteNew/Deep_content_tree-16             73.0ms ± 1%    74.2ms ± 1%     ~     (p=0.114 n=4+4)
SiteNew/Many_HTML_templates-16           37.9ms ± 0%    38.1ms ± 1%     ~     (p=0.114 n=4+4)
SiteNew/Page_collections-16              53.6ms ± 1%    54.7ms ± 1%   +2.09%  (p=0.029 n=4+4)

name                                   old alloc/op   new alloc/op   delta
SiteNew/Bundle_with_image/Edit-16         486kB ± 0%     430kB ± 0%  -11.47%  (p=0.029 n=4+4)
SiteNew/Bundle_with_JSON_file/Edit-16     265kB ± 0%     209kB ± 0%  -21.06%  (p=0.029 n=4+4)
SiteNew/Tags_and_categories/Edit-16      13.6MB ± 0%     8.8MB ± 0%  -34.93%  (p=0.029 n=4+4)
SiteNew/Canonify_URLs/Edit-16            66.5MB ± 0%    63.9MB ± 0%   -3.95%  (p=0.029 n=4+4)
SiteNew/Deep_content_tree/Edit-16        28.8MB ± 0%    25.8MB ± 0%  -10.55%  (p=0.029 n=4+4)
SiteNew/Many_HTML_templates/Edit-16      6.16MB ± 0%    5.56MB ± 0%   -9.86%  (p=0.029 n=4+4)
SiteNew/Page_collections/Edit-16         16.9MB ± 0%    16.0MB ± 0%   -5.19%  (p=0.029 n=4+4)
SiteNew/Bundle_with_image-16             2.28MB ± 0%    2.29MB ± 0%   +0.35%  (p=0.029 n=4+4)
SiteNew/Bundle_with_JSON_file-16         2.07MB ± 0%    2.07MB ± 0%     ~     (p=0.114 n=4+4)
SiteNew/Tags_and_categories-16           14.3MB ± 0%    13.2MB ± 0%   -7.30%  (p=0.029 n=4+4)
SiteNew/Canonify_URLs-16                 69.1MB ± 0%    69.0MB ± 0%     ~     (p=0.343 n=4+4)
SiteNew/Deep_content_tree-16             31.3MB ± 0%    31.8MB ± 0%   +1.49%  (p=0.029 n=4+4)
SiteNew/Many_HTML_templates-16           10.8MB ± 0%    10.9MB ± 0%   +1.11%  (p=0.029 n=4+4)
SiteNew/Page_collections-16              21.4MB ± 0%    21.6MB ± 0%   +1.15%  (p=0.029 n=4+4)

name                                   old allocs/op  new allocs/op  delta
SiteNew/Bundle_with_image/Edit-16         4.74k ± 0%     3.86k ± 0%  -18.57%  (p=0.029 n=4+4)
SiteNew/Bundle_with_JSON_file/Edit-16     4.73k ± 0%     3.85k ± 0%  -18.58%  (p=0.029 n=4+4)
SiteNew/Tags_and_categories/Edit-16        301k ± 0%      198k ± 0%  -34.14%  (p=0.029 n=4+4)
SiteNew/Canonify_URLs/Edit-16              389k ± 0%      373k ± 0%   -4.07%  (p=0.029 n=4+4)
SiteNew/Deep_content_tree/Edit-16          338k ± 0%      262k ± 0%  -22.63%  (p=0.029 n=4+4)
SiteNew/Many_HTML_templates/Edit-16        102k ± 0%       88k ± 0%  -13.81%  (p=0.029 n=4+4)
SiteNew/Page_collections/Edit-16           176k ± 0%      152k ± 0%  -13.32%  (p=0.029 n=4+4)
SiteNew/Bundle_with_image-16              26.8k ± 0%     26.8k ± 0%   +0.05%  (p=0.029 n=4+4)
SiteNew/Bundle_with_JSON_file-16          26.8k ± 0%     26.8k ± 0%   +0.05%  (p=0.029 n=4+4)
SiteNew/Tags_and_categories-16             273k ± 0%      245k ± 0%  -10.36%  (p=0.029 n=4+4)
SiteNew/Canonify_URLs-16                   396k ± 0%      398k ± 0%   +0.39%  (p=0.029 n=4+4)
SiteNew/Deep_content_tree-16               317k ± 0%      325k ± 0%   +2.53%  (p=0.029 n=4+4)
SiteNew/Many_HTML_templates-16             146k ± 0%      147k ± 0%   +0.98%  (p=0.029 n=4+4)
SiteNew/Page_collections-16                210k ± 0%      215k ± 0%   +2.44%  (p=0.029 n=4+4)
```

Fixes #6312
Fixes #6087
Fixes #6738
Fixes #6412
Fixes #6743
Fixes #6875
Fixes #6034
Fixes #6902
Fixes #6173
Fixes #6590
2020-02-18 09:49:42 +01:00
.circleci Add "go mod verify" to build scripts 2020-02-16 00:15:58 +01:00
.github Squashed 'docs/' changes from 54f0e8776..bd0e15bb6 2020-01-05 11:13:09 +01:00
bufferpool tests: Convert from testify to quicktest 2019-08-12 13:26:32 +02:00
cache resources/images: Make the image cache more robust 2019-11-25 18:59:06 +01:00
codegen tests: Convert from testify to quicktest 2019-08-12 13:26:32 +02:00
commands Introduce a tree map for all content 2020-02-18 09:49:42 +01:00
common Introduce a tree map for all content 2020-02-18 09:49:42 +01:00
compare tests: Convert from testify to quicktest 2019-08-12 13:26:32 +02:00
config Introduce a tree map for all content 2020-02-18 09:49:42 +01:00
create Introduce a tree map for all content 2020-02-18 09:49:42 +01:00
deploy deploy: Add ability to invalidate Google Cloud CDN 2019-10-03 16:53:45 +02:00
deps deps: Make the build flags shared between sites 2020-01-23 20:11:53 +01:00
docs releaser: Add release notes to /docs for release of 0.64.1 2020-02-09 20:44:34 +00:00
docshelper Add some missing doc comments 2017-08-03 15:57:51 +02:00
examples examples: Fix blog not building 2020-01-19 14:42:27 +01:00
helpers Introduce a tree map for all content 2020-02-18 09:49:42 +01:00
htesting Introduce a tree map for all content 2020-02-18 09:49:42 +01:00
hugofs Introduce a tree map for all content 2020-02-18 09:49:42 +01:00
hugolib Introduce a tree map for all content 2020-02-18 09:49:42 +01:00
identity Add render template hooks for links and images 2019-12-18 11:44:40 +01:00
langs Rework template handling for function and map lookups 2019-12-12 10:04:35 +01:00
lazy resources/images: Make the image cache more robust 2019-11-25 18:59:06 +01:00
livereload Fix livereload for @import case 2019-07-22 13:54:24 +02:00
markup markup/highlight: Fix chroma highlight 2020-02-17 14:59:26 +01:00
media Fix incorrect MIME type from image/jpg to image/jpeg 2019-12-19 18:02:11 +01:00
metrics tests: Convert from testify to quicktest 2019-08-12 13:26:32 +02:00
minifiers Revert to minify v2.6.1 2020-01-23 17:03:23 +01:00
modules modules: Fix "hugo mod get -u" with no arguments 2020-01-30 17:15:35 +01:00
navigation Fix Params case handling in the index, sort and where func 2019-11-22 18:41:50 +01:00
output And now finally fix the 404 templates 2020-01-27 12:30:31 +01:00
parser Introduce a tree map for all content 2020-02-18 09:49:42 +01:00
publisher Make Page an interface 2019-03-23 18:51:22 +01:00
related tests: Convert from testify to quicktest 2019-08-12 13:26:32 +02:00
releaser Update releasenotes_writer.go 2020-01-07 13:15:22 +01:00
resources Introduce a tree map for all content 2020-02-18 09:49:42 +01:00
scripts/fork_go_templates tpl: Get rid of the custom template truth logic 2019-12-18 20:05:22 +01:00
snap releaser: Prepare repository for 0.65.0-DEV 2020-02-09 20:53:18 +00:00
source source: Remove debug check left during development 2019-08-23 07:37:56 +02:00
tpl Introduce a tree map for all content 2020-02-18 09:49:42 +01:00
transform transform/livereloadinject: Add defer to livereload script tag 2020-01-31 08:15:22 +01:00
watcher Add some missing doc comments 2017-08-03 15:57:51 +02:00
.dockerignore Support Docker args TAGS, WORKDIR, CGO; speed up repetitive builds 2019-02-22 11:53:17 -08:00
.gitattributes .gitattributes: Exclude *.svg from CRLF/LF conversion 2018-07-09 21:06:15 -06:00
.gitignore Create lightweight forks of text/template and html/template 2019-12-12 09:59:34 +01:00
.gitmodules Remove the theme submodule from /docs 2017-08-10 14:54:19 +02:00
.mailmap Add .mailmap to get a more correct author log 2015-01-28 16:50:36 +01:00
.travis.yml Add "go mod verify" to build scripts 2020-02-16 00:15:58 +01:00
CONTRIBUTING.md CONTRIBUTING: Fix note about CGO 2020-02-06 09:09:08 +01:00
Dockerfile Add git to Dockerfile 2020-02-14 16:49:20 +01:00
LICENSE Add a GitHub compatible Apache 2 license text 2018-04-16 00:03:30 +02:00
README.md Update Go version requirement 2020-02-04 23:48:31 +01:00
bench.sh Add GOEXE to support building with different versions of `go` 2017-07-16 00:35:15 +02:00
benchSite.sh Run benchmarks 3 times 2018-01-14 00:49:01 +01:00
benchbep.sh Add Hugo Modules 2019-07-24 09:35:53 +02:00
bepdock.sh build: Update the temp docker script 2018-09-24 12:41:12 +02:00
go.mod deps: Update golibsass 2020-02-12 18:56:47 +01:00
go.sum Introduce a tree map for all content 2020-02-18 09:49:42 +01:00
goreleaser.yml releaser: Update Go Releaser and consolidate the 2 configs 2019-06-11 00:02:06 +02:00
magefile.go Create lightweight forks of text/template and html/template 2019-12-12 09:59:34 +01:00
main.go Add HUGO_NUMWORKERMULTIPLIER 2019-04-05 10:21:25 +02:00
pull-docs.sh Allow the pull-docs script to pull other than master 2017-09-23 10:13:40 +02:00
requirements.txt Squashed 'docs/' changes from 51c4f3184..af4b7ac5b 2019-12-15 10:35:09 +01:00

README.md

Hugo

A Fast and Flexible Static Site Generator built with love by bep, spf13 and friends in Go.

Website | Forum | Documentation | Installation Guide | Contribution Guide | Twitter

GoDoc Linux and macOS Build Status Go Report Card

Overview

Hugo is a static HTML and CSS website generator written in Go. It is optimized for speed, ease of use, and configurability. Hugo takes a directory with content and templates and renders them into a full HTML website.

Hugo relies on Markdown files with front matter for metadata, and you can run Hugo from any directory. This works well for shared hosts and other systems where you dont have a privileged account.

Hugo renders a typical website of moderate size in a fraction of a second. A good rule of thumb is that each piece of content renders in around 1 millisecond.

Hugo is designed to work well for any kind of website including blogs, tumbles, and docs.

Supported Architectures

Currently, we provide pre-built Hugo binaries for Windows, Linux, FreeBSD, NetBSD, DragonFly BSD, Open BSD, macOS (Darwin), and Android for x64, i386 and ARM architectures.

Hugo may also be compiled from source wherever the Go compiler tool chain can run, e.g. for other operating systems including Plan 9 and Solaris.

Complete documentation is available at Hugo Documentation.

Choose How to Install

If you want to use Hugo as your site generator, simply install the Hugo binaries. The Hugo binaries have no external dependencies.

To contribute to the Hugo source code or documentation, you should fork the Hugo GitHub project and clone it to your local machine.

Finally, you can install the Hugo source code with go, build the binaries yourself, and run Hugo that way. Building the binaries is an easy task for an experienced go getter.

Install Hugo as Your Site Generator (Binary Install)

Use the installation instructions in the Hugo documentation.

Build and Install the Binaries from Source (Advanced Install)

Prerequisite Tools

Fetch from GitHub

Since Hugo 0.48, Hugo uses the Go Modules support built into Go 1.11 to build. The easiest is to clone Hugo in a directory outside of GOPATH, as in the following example:

mkdir $HOME/src
cd $HOME/src
git clone https://github.com/gohugoio/hugo.git
cd hugo
go install

If you are a Windows user, substitute the $HOME environment variable above with %USERPROFILE%.

The Hugo Documentation

The Hugo documentation now lives in its own repository, see https://github.com/gohugoio/hugoDocs. But we do keep a version of that documentation as a git subtree in this repository. To build the sub folder /docs as a Hugo site, you need to clone this repo:

git clone git@github.com:gohugoio/hugo.git

Contributing to Hugo

For a complete guide to contributing to Hugo, see the Contribution Guide.

We welcome contributions to Hugo of any kind including documentation, themes, organization, tutorials, blog posts, bug reports, issues, feature requests, feature implementations, pull requests, answering questions on the forum, helping to manage issues, etc.

The Hugo community and maintainers are very active and helpful, and the project benefits greatly from this activity.

Asking Support Questions

We have an active discussion forum where users and developers can ask questions. Please don't use the GitHub issue tracker to ask questions.

Reporting Issues

If you believe you have found a defect in Hugo or its documentation, use the GitHub issue tracker to report the problem to the Hugo maintainers. If you're not sure if it's a bug or not, start by asking in the discussion forum. When reporting the issue, please provide the version of Hugo in use (hugo version).

Submitting Patches

The Hugo project welcomes all contributors and contributions regardless of skill or experience level. If you are interested in helping with the project, we will help you with your contribution. Hugo is a very active project with many contributions happening daily.

We want to create the best possible product for our users and the best contribution experience for our developers, we have a set of guidelines which ensure that all contributions are acceptable. The guidelines are not intended as a filter or barrier to participation. If you are unfamiliar with the contribution process, the Hugo team will help you and teach you how to bring your contribution in accordance with the guidelines.

For a complete guide to contributing code to Hugo, see the Contribution Guide.

Analytics

Dependencies

Hugo stands on the shoulder of many great open source libraries, in lexical order:

Dependency License
github.com/BurntSushi/locker The Unlicense
github.com/BurntSushi/toml MIT License
github.com/PuerkitoBio/purell BSD 3-Clause "New" or "Revised" License
github.com/PuerkitoBio/urlesc BSD 3-Clause "New" or "Revised" License
github.com/alecthomas/chroma MIT License
github.com/bep/debounce MIT License
github.com/bep/gitmap MIT License
github.com/bep/go-tocss MIT License
github.com/niklasfasching/go-org MIT License
github.com/cpuguy83/go-md2man MIT License
github.com/danwakefield/fnmatch BSD 2-Clause "Simplified" License
github.com/disintegration/imaging MIT License
github.com/dlclark/regexp2 MIT License
github.com/eknkc/amber MIT License
github.com/fsnotify/fsnotify BSD 3-Clause "New" or "Revised" License
github.com/gobwas/glob MIT License
github.com/gorilla/websocket BSD 2-Clause "Simplified" License
github.com/hashicorp/go-immutable-radix Mozilla Public License 2.0
github.com/hashicorp/golang-lru Mozilla Public License 2.0
github.com/hashicorp/hcl Mozilla Public License 2.0
github.com/jdkato/prose MIT License
github.com/kyokomi/emoji MIT License
github.com/magiconair/properties BSD 2-Clause "Simplified" License
github.com/markbates/inflect MIT License
github.com/mattn/go-isatty MIT License
github.com/mattn/go-runewidth MIT License
github.com/miekg/mmark Simplified BSD License
github.com/mitchellh/hashstructure MIT License
github.com/mitchellh/mapstructure MIT License
github.com/muesli/smartcrop MIT License
github.com/nicksnyder/go-i18n MIT License
github.com/olekukonko/tablewriter MIT License
github.com/pelletier/go-toml MIT License
github.com/pkg/errors BSD 2-Clause "Simplified" License
github.com/russross/blackfriday Simplified BSD License
github.com/shurcooL/sanitized_anchor_name MIT License
github.com/spf13/afero Apache License 2.0
github.com/spf13/cast MIT License
github.com/spf13/cobra Apache License 2.0
github.com/spf13/fsync MIT License
github.com/spf13/jwalterweatherman MIT License
github.com/spf13/nitro Apache License 2.0
github.com/spf13/pflag BSD 3-Clause "New" or "Revised" License
github.com/spf13/viper MIT License
github.com/tdewolff/minify MIT License
github.com/tdewolff/parse MIT License
github.com/wellington/go-libsass Apache License 2.0
github.com/yosssi/ace MIT License
golang.org/x/image BSD 3-Clause "New" or "Revised" License
golang.org/x/net BSD 3-Clause "New" or "Revised" License
golang.org/x/sync BSD 3-Clause "New" or "Revised" License
golang.org/x/sys BSD 3-Clause "New" or "Revised" License
golang.org/x/text BSD 3-Clause "New" or "Revised" License
gopkg.in/yaml.v2 Apache License 2.0