mirror of https://github.com/git/git.git
Merge branch 'ds/maintenance-part-2'
"git maintenance", an extended big brother of "git gc", continues to evolve. * ds/maintenance-part-2: maintenance: add incremental-repack auto condition maintenance: auto-size incremental-repack batch maintenance: add incremental-repack task midx: use start_delayed_progress() midx: enable core.multiPackIndex by default maintenance: create auto condition for loose-objects maintenance: add loose-objects task maintenance: add prefetch task
This commit is contained in:
commit
52b8c8c716
|
@ -606,8 +606,8 @@ core.useReplaceRefs::
|
||||||
|
|
||||||
core.multiPackIndex::
|
core.multiPackIndex::
|
||||||
Use the multi-pack-index file to track multiple packfiles using a
|
Use the multi-pack-index file to track multiple packfiles using a
|
||||||
single index. See link:technical/multi-pack-index.html[the
|
single index. See linkgit:git-multi-pack-index[1] for more
|
||||||
multi-pack-index design document].
|
information. Defaults to true.
|
||||||
|
|
||||||
core.sparseCheckout::
|
core.sparseCheckout::
|
||||||
Enable "sparse checkout" feature. See linkgit:git-sparse-checkout[1]
|
Enable "sparse checkout" feature. See linkgit:git-sparse-checkout[1]
|
||||||
|
|
|
@ -14,3 +14,21 @@ maintenance.commit-graph.auto::
|
||||||
reachable commits that are not in the commit-graph file is at least
|
reachable commits that are not in the commit-graph file is at least
|
||||||
the value of `maintenance.commit-graph.auto`. The default value is
|
the value of `maintenance.commit-graph.auto`. The default value is
|
||||||
100.
|
100.
|
||||||
|
|
||||||
|
maintenance.loose-objects.auto::
|
||||||
|
This integer config option controls how often the `loose-objects` task
|
||||||
|
should be run as part of `git maintenance run --auto`. If zero, then
|
||||||
|
the `loose-objects` task will not run with the `--auto` option. A
|
||||||
|
negative value will force the task to run every time. Otherwise, a
|
||||||
|
positive value implies the command should run when the number of
|
||||||
|
loose objects is at least the value of `maintenance.loose-objects.auto`.
|
||||||
|
The default value is 100.
|
||||||
|
|
||||||
|
maintenance.incremental-repack.auto::
|
||||||
|
This integer config option controls how often the `incremental-repack`
|
||||||
|
task should be run as part of `git maintenance run --auto`. If zero,
|
||||||
|
then the `incremental-repack` task will not run with the `--auto`
|
||||||
|
option. A negative value will force the task to run every time.
|
||||||
|
Otherwise, a positive value implies the command should run when the
|
||||||
|
number of pack-files not in the multi-pack-index is at least the value
|
||||||
|
of `maintenance.incremental-repack.auto`. The default value is 10.
|
||||||
|
|
|
@ -47,6 +47,21 @@ commit-graph::
|
||||||
`commit-graph-chain` file. They will be deleted by a later run based
|
`commit-graph-chain` file. They will be deleted by a later run based
|
||||||
on the expiration delay.
|
on the expiration delay.
|
||||||
|
|
||||||
|
prefetch::
|
||||||
|
The `prefetch` task updates the object directory with the latest
|
||||||
|
objects from all registered remotes. For each remote, a `git fetch`
|
||||||
|
command is run. The refmap is custom to avoid updating local or remote
|
||||||
|
branches (those in `refs/heads` or `refs/remotes`). Instead, the
|
||||||
|
remote refs are stored in `refs/prefetch/<remote>/`. Also, tags are
|
||||||
|
not updated.
|
||||||
|
+
|
||||||
|
This is done to avoid disrupting the remote-tracking branches. The end users
|
||||||
|
expect these refs to stay unmoved unless they initiate a fetch. With prefetch
|
||||||
|
task, however, the objects necessary to complete a later real fetch would
|
||||||
|
already be obtained, so the real fetch would go faster. In the ideal case,
|
||||||
|
it will just become an update to bunch of remote-tracking branches without
|
||||||
|
any object transfer.
|
||||||
|
|
||||||
gc::
|
gc::
|
||||||
Clean up unnecessary files and optimize the local repository. "GC"
|
Clean up unnecessary files and optimize the local repository. "GC"
|
||||||
stands for "garbage collection," but this task performs many
|
stands for "garbage collection," but this task performs many
|
||||||
|
@ -55,6 +70,39 @@ gc::
|
||||||
be disruptive in some situations, as it deletes stale data. See
|
be disruptive in some situations, as it deletes stale data. See
|
||||||
linkgit:git-gc[1] for more details on garbage collection in Git.
|
linkgit:git-gc[1] for more details on garbage collection in Git.
|
||||||
|
|
||||||
|
loose-objects::
|
||||||
|
The `loose-objects` job cleans up loose objects and places them into
|
||||||
|
pack-files. In order to prevent race conditions with concurrent Git
|
||||||
|
commands, it follows a two-step process. First, it deletes any loose
|
||||||
|
objects that already exist in a pack-file; concurrent Git processes
|
||||||
|
will examine the pack-file for the object data instead of the loose
|
||||||
|
object. Second, it creates a new pack-file (starting with "loose-")
|
||||||
|
containing a batch of loose objects. The batch size is limited to 50
|
||||||
|
thousand objects to prevent the job from taking too long on a
|
||||||
|
repository with many loose objects. The `gc` task writes unreachable
|
||||||
|
objects as loose objects to be cleaned up by a later step only if
|
||||||
|
they are not re-added to a pack-file; for this reason it is not
|
||||||
|
advisable to enable both the `loose-objects` and `gc` tasks at the
|
||||||
|
same time.
|
||||||
|
|
||||||
|
incremental-repack::
|
||||||
|
The `incremental-repack` job repacks the object directory
|
||||||
|
using the `multi-pack-index` feature. In order to prevent race
|
||||||
|
conditions with concurrent Git commands, it follows a two-step
|
||||||
|
process. First, it calls `git multi-pack-index expire` to delete
|
||||||
|
pack-files unreferenced by the `multi-pack-index` file. Second, it
|
||||||
|
calls `git multi-pack-index repack` to select several small
|
||||||
|
pack-files and repack them into a bigger one, and then update the
|
||||||
|
`multi-pack-index` entries that refer to the small pack-files to
|
||||||
|
refer to the new pack-file. This prepares those small pack-files
|
||||||
|
for deletion upon the next run of `git multi-pack-index expire`.
|
||||||
|
The selection of the small pack-files is such that the expected
|
||||||
|
size of the big pack-file is at least the batch size; see the
|
||||||
|
`--batch-size` option for the `repack` subcommand in
|
||||||
|
linkgit:git-multi-pack-index[1]. The default batch-size is zero,
|
||||||
|
which is a special case that attempts to repack all pack-files
|
||||||
|
into a single pack-file.
|
||||||
|
|
||||||
OPTIONS
|
OPTIONS
|
||||||
-------
|
-------
|
||||||
--auto::
|
--auto::
|
||||||
|
|
326
builtin/gc.c
326
builtin/gc.c
|
@ -29,6 +29,8 @@
|
||||||
#include "tree.h"
|
#include "tree.h"
|
||||||
#include "promisor-remote.h"
|
#include "promisor-remote.h"
|
||||||
#include "refs.h"
|
#include "refs.h"
|
||||||
|
#include "remote.h"
|
||||||
|
#include "object-store.h"
|
||||||
|
|
||||||
#define FAILED_RUN "failed to run %s"
|
#define FAILED_RUN "failed to run %s"
|
||||||
|
|
||||||
|
@ -816,6 +818,51 @@ static int maintenance_task_commit_graph(struct maintenance_run_opts *opts)
|
||||||
return 0;
|
return 0;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
static int fetch_remote(const char *remote, struct maintenance_run_opts *opts)
|
||||||
|
{
|
||||||
|
struct child_process child = CHILD_PROCESS_INIT;
|
||||||
|
|
||||||
|
child.git_cmd = 1;
|
||||||
|
strvec_pushl(&child.args, "fetch", remote, "--prune", "--no-tags",
|
||||||
|
"--no-write-fetch-head", "--recurse-submodules=no",
|
||||||
|
"--refmap=", NULL);
|
||||||
|
|
||||||
|
if (opts->quiet)
|
||||||
|
strvec_push(&child.args, "--quiet");
|
||||||
|
|
||||||
|
strvec_pushf(&child.args, "+refs/heads/*:refs/prefetch/%s/*", remote);
|
||||||
|
|
||||||
|
return !!run_command(&child);
|
||||||
|
}
|
||||||
|
|
||||||
|
static int append_remote(struct remote *remote, void *cbdata)
|
||||||
|
{
|
||||||
|
struct string_list *remotes = (struct string_list *)cbdata;
|
||||||
|
|
||||||
|
string_list_append(remotes, remote->name);
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
static int maintenance_task_prefetch(struct maintenance_run_opts *opts)
|
||||||
|
{
|
||||||
|
int result = 0;
|
||||||
|
struct string_list_item *item;
|
||||||
|
struct string_list remotes = STRING_LIST_INIT_DUP;
|
||||||
|
|
||||||
|
if (for_each_remote(append_remote, &remotes)) {
|
||||||
|
error(_("failed to fill remotes"));
|
||||||
|
result = 1;
|
||||||
|
goto cleanup;
|
||||||
|
}
|
||||||
|
|
||||||
|
for_each_string_list_item(item, &remotes)
|
||||||
|
result |= fetch_remote(item->string, opts);
|
||||||
|
|
||||||
|
cleanup:
|
||||||
|
string_list_clear(&remotes, 0);
|
||||||
|
return result;
|
||||||
|
}
|
||||||
|
|
||||||
static int maintenance_task_gc(struct maintenance_run_opts *opts)
|
static int maintenance_task_gc(struct maintenance_run_opts *opts)
|
||||||
{
|
{
|
||||||
struct child_process child = CHILD_PROCESS_INIT;
|
struct child_process child = CHILD_PROCESS_INIT;
|
||||||
|
@ -834,6 +881,268 @@ static int maintenance_task_gc(struct maintenance_run_opts *opts)
|
||||||
return run_command(&child);
|
return run_command(&child);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
static int prune_packed(struct maintenance_run_opts *opts)
|
||||||
|
{
|
||||||
|
struct child_process child = CHILD_PROCESS_INIT;
|
||||||
|
|
||||||
|
child.git_cmd = 1;
|
||||||
|
strvec_push(&child.args, "prune-packed");
|
||||||
|
|
||||||
|
if (opts->quiet)
|
||||||
|
strvec_push(&child.args, "--quiet");
|
||||||
|
|
||||||
|
return !!run_command(&child);
|
||||||
|
}
|
||||||
|
|
||||||
|
struct write_loose_object_data {
|
||||||
|
FILE *in;
|
||||||
|
int count;
|
||||||
|
int batch_size;
|
||||||
|
};
|
||||||
|
|
||||||
|
static int loose_object_auto_limit = 100;
|
||||||
|
|
||||||
|
static int loose_object_count(const struct object_id *oid,
|
||||||
|
const char *path,
|
||||||
|
void *data)
|
||||||
|
{
|
||||||
|
int *count = (int*)data;
|
||||||
|
if (++(*count) >= loose_object_auto_limit)
|
||||||
|
return 1;
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
static int loose_object_auto_condition(void)
|
||||||
|
{
|
||||||
|
int count = 0;
|
||||||
|
|
||||||
|
git_config_get_int("maintenance.loose-objects.auto",
|
||||||
|
&loose_object_auto_limit);
|
||||||
|
|
||||||
|
if (!loose_object_auto_limit)
|
||||||
|
return 0;
|
||||||
|
if (loose_object_auto_limit < 0)
|
||||||
|
return 1;
|
||||||
|
|
||||||
|
return for_each_loose_file_in_objdir(the_repository->objects->odb->path,
|
||||||
|
loose_object_count,
|
||||||
|
NULL, NULL, &count);
|
||||||
|
}
|
||||||
|
|
||||||
|
static int bail_on_loose(const struct object_id *oid,
|
||||||
|
const char *path,
|
||||||
|
void *data)
|
||||||
|
{
|
||||||
|
return 1;
|
||||||
|
}
|
||||||
|
|
||||||
|
static int write_loose_object_to_stdin(const struct object_id *oid,
|
||||||
|
const char *path,
|
||||||
|
void *data)
|
||||||
|
{
|
||||||
|
struct write_loose_object_data *d = (struct write_loose_object_data *)data;
|
||||||
|
|
||||||
|
fprintf(d->in, "%s\n", oid_to_hex(oid));
|
||||||
|
|
||||||
|
return ++(d->count) > d->batch_size;
|
||||||
|
}
|
||||||
|
|
||||||
|
static int pack_loose(struct maintenance_run_opts *opts)
|
||||||
|
{
|
||||||
|
struct repository *r = the_repository;
|
||||||
|
int result = 0;
|
||||||
|
struct write_loose_object_data data;
|
||||||
|
struct child_process pack_proc = CHILD_PROCESS_INIT;
|
||||||
|
|
||||||
|
/*
|
||||||
|
* Do not start pack-objects process
|
||||||
|
* if there are no loose objects.
|
||||||
|
*/
|
||||||
|
if (!for_each_loose_file_in_objdir(r->objects->odb->path,
|
||||||
|
bail_on_loose,
|
||||||
|
NULL, NULL, NULL))
|
||||||
|
return 0;
|
||||||
|
|
||||||
|
pack_proc.git_cmd = 1;
|
||||||
|
|
||||||
|
strvec_push(&pack_proc.args, "pack-objects");
|
||||||
|
if (opts->quiet)
|
||||||
|
strvec_push(&pack_proc.args, "--quiet");
|
||||||
|
strvec_pushf(&pack_proc.args, "%s/pack/loose", r->objects->odb->path);
|
||||||
|
|
||||||
|
pack_proc.in = -1;
|
||||||
|
|
||||||
|
if (start_command(&pack_proc)) {
|
||||||
|
error(_("failed to start 'git pack-objects' process"));
|
||||||
|
return 1;
|
||||||
|
}
|
||||||
|
|
||||||
|
data.in = xfdopen(pack_proc.in, "w");
|
||||||
|
data.count = 0;
|
||||||
|
data.batch_size = 50000;
|
||||||
|
|
||||||
|
for_each_loose_file_in_objdir(r->objects->odb->path,
|
||||||
|
write_loose_object_to_stdin,
|
||||||
|
NULL,
|
||||||
|
NULL,
|
||||||
|
&data);
|
||||||
|
|
||||||
|
fclose(data.in);
|
||||||
|
|
||||||
|
if (finish_command(&pack_proc)) {
|
||||||
|
error(_("failed to finish 'git pack-objects' process"));
|
||||||
|
result = 1;
|
||||||
|
}
|
||||||
|
|
||||||
|
return result;
|
||||||
|
}
|
||||||
|
|
||||||
|
static int maintenance_task_loose_objects(struct maintenance_run_opts *opts)
|
||||||
|
{
|
||||||
|
return prune_packed(opts) || pack_loose(opts);
|
||||||
|
}
|
||||||
|
|
||||||
|
static int incremental_repack_auto_condition(void)
|
||||||
|
{
|
||||||
|
struct packed_git *p;
|
||||||
|
int enabled;
|
||||||
|
int incremental_repack_auto_limit = 10;
|
||||||
|
int count = 0;
|
||||||
|
|
||||||
|
if (git_config_get_bool("core.multiPackIndex", &enabled) ||
|
||||||
|
!enabled)
|
||||||
|
return 0;
|
||||||
|
|
||||||
|
git_config_get_int("maintenance.incremental-repack.auto",
|
||||||
|
&incremental_repack_auto_limit);
|
||||||
|
|
||||||
|
if (!incremental_repack_auto_limit)
|
||||||
|
return 0;
|
||||||
|
if (incremental_repack_auto_limit < 0)
|
||||||
|
return 1;
|
||||||
|
|
||||||
|
for (p = get_packed_git(the_repository);
|
||||||
|
count < incremental_repack_auto_limit && p;
|
||||||
|
p = p->next) {
|
||||||
|
if (!p->multi_pack_index)
|
||||||
|
count++;
|
||||||
|
}
|
||||||
|
|
||||||
|
return count >= incremental_repack_auto_limit;
|
||||||
|
}
|
||||||
|
|
||||||
|
static int multi_pack_index_write(struct maintenance_run_opts *opts)
|
||||||
|
{
|
||||||
|
struct child_process child = CHILD_PROCESS_INIT;
|
||||||
|
|
||||||
|
child.git_cmd = 1;
|
||||||
|
strvec_pushl(&child.args, "multi-pack-index", "write", NULL);
|
||||||
|
|
||||||
|
if (opts->quiet)
|
||||||
|
strvec_push(&child.args, "--no-progress");
|
||||||
|
|
||||||
|
if (run_command(&child))
|
||||||
|
return error(_("failed to write multi-pack-index"));
|
||||||
|
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
static int multi_pack_index_expire(struct maintenance_run_opts *opts)
|
||||||
|
{
|
||||||
|
struct child_process child = CHILD_PROCESS_INIT;
|
||||||
|
|
||||||
|
child.git_cmd = 1;
|
||||||
|
strvec_pushl(&child.args, "multi-pack-index", "expire", NULL);
|
||||||
|
|
||||||
|
if (opts->quiet)
|
||||||
|
strvec_push(&child.args, "--no-progress");
|
||||||
|
|
||||||
|
close_object_store(the_repository->objects);
|
||||||
|
|
||||||
|
if (run_command(&child))
|
||||||
|
return error(_("'git multi-pack-index expire' failed"));
|
||||||
|
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
#define TWO_GIGABYTES (INT32_MAX)
|
||||||
|
|
||||||
|
static off_t get_auto_pack_size(void)
|
||||||
|
{
|
||||||
|
/*
|
||||||
|
* The "auto" value is special: we optimize for
|
||||||
|
* one large pack-file (i.e. from a clone) and
|
||||||
|
* expect the rest to be small and they can be
|
||||||
|
* repacked quickly.
|
||||||
|
*
|
||||||
|
* The strategy we select here is to select a
|
||||||
|
* size that is one more than the second largest
|
||||||
|
* pack-file. This ensures that we will repack
|
||||||
|
* at least two packs if there are three or more
|
||||||
|
* packs.
|
||||||
|
*/
|
||||||
|
off_t max_size = 0;
|
||||||
|
off_t second_largest_size = 0;
|
||||||
|
off_t result_size;
|
||||||
|
struct packed_git *p;
|
||||||
|
struct repository *r = the_repository;
|
||||||
|
|
||||||
|
reprepare_packed_git(r);
|
||||||
|
for (p = get_all_packs(r); p; p = p->next) {
|
||||||
|
if (p->pack_size > max_size) {
|
||||||
|
second_largest_size = max_size;
|
||||||
|
max_size = p->pack_size;
|
||||||
|
} else if (p->pack_size > second_largest_size)
|
||||||
|
second_largest_size = p->pack_size;
|
||||||
|
}
|
||||||
|
|
||||||
|
result_size = second_largest_size + 1;
|
||||||
|
|
||||||
|
/* But limit ourselves to a batch size of 2g */
|
||||||
|
if (result_size > TWO_GIGABYTES)
|
||||||
|
result_size = TWO_GIGABYTES;
|
||||||
|
|
||||||
|
return result_size;
|
||||||
|
}
|
||||||
|
|
||||||
|
static int multi_pack_index_repack(struct maintenance_run_opts *opts)
|
||||||
|
{
|
||||||
|
struct child_process child = CHILD_PROCESS_INIT;
|
||||||
|
|
||||||
|
child.git_cmd = 1;
|
||||||
|
strvec_pushl(&child.args, "multi-pack-index", "repack", NULL);
|
||||||
|
|
||||||
|
if (opts->quiet)
|
||||||
|
strvec_push(&child.args, "--no-progress");
|
||||||
|
|
||||||
|
strvec_pushf(&child.args, "--batch-size=%"PRIuMAX,
|
||||||
|
(uintmax_t)get_auto_pack_size());
|
||||||
|
|
||||||
|
close_object_store(the_repository->objects);
|
||||||
|
|
||||||
|
if (run_command(&child))
|
||||||
|
return error(_("'git multi-pack-index repack' failed"));
|
||||||
|
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
static int maintenance_task_incremental_repack(struct maintenance_run_opts *opts)
|
||||||
|
{
|
||||||
|
prepare_repo_settings(the_repository);
|
||||||
|
if (!the_repository->settings.core_multi_pack_index) {
|
||||||
|
warning(_("skipping incremental-repack task because core.multiPackIndex is disabled"));
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (multi_pack_index_write(opts))
|
||||||
|
return 1;
|
||||||
|
if (multi_pack_index_expire(opts))
|
||||||
|
return 1;
|
||||||
|
if (multi_pack_index_repack(opts))
|
||||||
|
return 1;
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
|
||||||
typedef int maintenance_task_fn(struct maintenance_run_opts *opts);
|
typedef int maintenance_task_fn(struct maintenance_run_opts *opts);
|
||||||
|
|
||||||
/*
|
/*
|
||||||
|
@ -854,6 +1163,9 @@ struct maintenance_task {
|
||||||
};
|
};
|
||||||
|
|
||||||
enum maintenance_task_label {
|
enum maintenance_task_label {
|
||||||
|
TASK_PREFETCH,
|
||||||
|
TASK_LOOSE_OBJECTS,
|
||||||
|
TASK_INCREMENTAL_REPACK,
|
||||||
TASK_GC,
|
TASK_GC,
|
||||||
TASK_COMMIT_GRAPH,
|
TASK_COMMIT_GRAPH,
|
||||||
|
|
||||||
|
@ -862,6 +1174,20 @@ enum maintenance_task_label {
|
||||||
};
|
};
|
||||||
|
|
||||||
static struct maintenance_task tasks[] = {
|
static struct maintenance_task tasks[] = {
|
||||||
|
[TASK_PREFETCH] = {
|
||||||
|
"prefetch",
|
||||||
|
maintenance_task_prefetch,
|
||||||
|
},
|
||||||
|
[TASK_LOOSE_OBJECTS] = {
|
||||||
|
"loose-objects",
|
||||||
|
maintenance_task_loose_objects,
|
||||||
|
loose_object_auto_condition,
|
||||||
|
},
|
||||||
|
[TASK_INCREMENTAL_REPACK] = {
|
||||||
|
"incremental-repack",
|
||||||
|
maintenance_task_incremental_repack,
|
||||||
|
incremental_repack_auto_condition,
|
||||||
|
},
|
||||||
[TASK_GC] = {
|
[TASK_GC] = {
|
||||||
"gc",
|
"gc",
|
||||||
maintenance_task_gc,
|
maintenance_task_gc,
|
||||||
|
|
21
midx.c
21
midx.c
|
@ -10,6 +10,7 @@
|
||||||
#include "progress.h"
|
#include "progress.h"
|
||||||
#include "trace2.h"
|
#include "trace2.h"
|
||||||
#include "run-command.h"
|
#include "run-command.h"
|
||||||
|
#include "repository.h"
|
||||||
|
|
||||||
#define MIDX_SIGNATURE 0x4d494458 /* "MIDX" */
|
#define MIDX_SIGNATURE 0x4d494458 /* "MIDX" */
|
||||||
#define MIDX_VERSION 1
|
#define MIDX_VERSION 1
|
||||||
|
@ -398,15 +399,9 @@ int prepare_multi_pack_index_one(struct repository *r, const char *object_dir, i
|
||||||
{
|
{
|
||||||
struct multi_pack_index *m;
|
struct multi_pack_index *m;
|
||||||
struct multi_pack_index *m_search;
|
struct multi_pack_index *m_search;
|
||||||
int config_value;
|
|
||||||
static int env_value = -1;
|
|
||||||
|
|
||||||
if (env_value < 0)
|
prepare_repo_settings(r);
|
||||||
env_value = git_env_bool(GIT_TEST_MULTI_PACK_INDEX, 0);
|
if (!r->settings.core_multi_pack_index)
|
||||||
|
|
||||||
if (!env_value &&
|
|
||||||
(repo_config_get_bool(r, "core.multipackindex", &config_value) ||
|
|
||||||
!config_value))
|
|
||||||
return 0;
|
return 0;
|
||||||
|
|
||||||
for (m_search = r->objects->multi_pack_index; m_search; m_search = m_search->next)
|
for (m_search = r->objects->multi_pack_index; m_search; m_search = m_search->next)
|
||||||
|
@ -850,7 +845,7 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *
|
||||||
|
|
||||||
packs.pack_paths_checked = 0;
|
packs.pack_paths_checked = 0;
|
||||||
if (flags & MIDX_PROGRESS)
|
if (flags & MIDX_PROGRESS)
|
||||||
packs.progress = start_progress(_("Adding packfiles to multi-pack-index"), 0);
|
packs.progress = start_delayed_progress(_("Adding packfiles to multi-pack-index"), 0);
|
||||||
else
|
else
|
||||||
packs.progress = NULL;
|
packs.progress = NULL;
|
||||||
|
|
||||||
|
@ -987,7 +982,7 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *
|
||||||
}
|
}
|
||||||
|
|
||||||
if (flags & MIDX_PROGRESS)
|
if (flags & MIDX_PROGRESS)
|
||||||
progress = start_progress(_("Writing chunks to multi-pack-index"),
|
progress = start_delayed_progress(_("Writing chunks to multi-pack-index"),
|
||||||
num_chunks);
|
num_chunks);
|
||||||
for (i = 0; i < num_chunks; i++) {
|
for (i = 0; i < num_chunks; i++) {
|
||||||
if (written != chunk_offsets[i])
|
if (written != chunk_offsets[i])
|
||||||
|
@ -1129,7 +1124,7 @@ int verify_midx_file(struct repository *r, const char *object_dir, unsigned flag
|
||||||
}
|
}
|
||||||
|
|
||||||
if (flags & MIDX_PROGRESS)
|
if (flags & MIDX_PROGRESS)
|
||||||
progress = start_progress(_("Looking for referenced packfiles"),
|
progress = start_delayed_progress(_("Looking for referenced packfiles"),
|
||||||
m->num_packs);
|
m->num_packs);
|
||||||
for (i = 0; i < m->num_packs; i++) {
|
for (i = 0; i < m->num_packs; i++) {
|
||||||
if (prepare_midx_pack(r, m, i))
|
if (prepare_midx_pack(r, m, i))
|
||||||
|
@ -1250,7 +1245,7 @@ int expire_midx_packs(struct repository *r, const char *object_dir, unsigned fla
|
||||||
count = xcalloc(m->num_packs, sizeof(uint32_t));
|
count = xcalloc(m->num_packs, sizeof(uint32_t));
|
||||||
|
|
||||||
if (flags & MIDX_PROGRESS)
|
if (flags & MIDX_PROGRESS)
|
||||||
progress = start_progress(_("Counting referenced objects"),
|
progress = start_delayed_progress(_("Counting referenced objects"),
|
||||||
m->num_objects);
|
m->num_objects);
|
||||||
for (i = 0; i < m->num_objects; i++) {
|
for (i = 0; i < m->num_objects; i++) {
|
||||||
int pack_int_id = nth_midxed_pack_int_id(m, i);
|
int pack_int_id = nth_midxed_pack_int_id(m, i);
|
||||||
|
@ -1260,7 +1255,7 @@ int expire_midx_packs(struct repository *r, const char *object_dir, unsigned fla
|
||||||
stop_progress(&progress);
|
stop_progress(&progress);
|
||||||
|
|
||||||
if (flags & MIDX_PROGRESS)
|
if (flags & MIDX_PROGRESS)
|
||||||
progress = start_progress(_("Finding and deleting unreferenced packfiles"),
|
progress = start_delayed_progress(_("Finding and deleting unreferenced packfiles"),
|
||||||
m->num_packs);
|
m->num_packs);
|
||||||
for (i = 0; i < m->num_packs; i++) {
|
for (i = 0; i < m->num_packs; i++) {
|
||||||
char *pack_name;
|
char *pack_name;
|
||||||
|
|
|
@ -1,6 +1,7 @@
|
||||||
#include "cache.h"
|
#include "cache.h"
|
||||||
#include "config.h"
|
#include "config.h"
|
||||||
#include "repository.h"
|
#include "repository.h"
|
||||||
|
#include "midx.h"
|
||||||
|
|
||||||
#define UPDATE_DEFAULT_BOOL(s,v) do { if (s == -1) { s = v; } } while(0)
|
#define UPDATE_DEFAULT_BOOL(s,v) do { if (s == -1) { s = v; } } while(0)
|
||||||
|
|
||||||
|
@ -52,6 +53,11 @@ void prepare_repo_settings(struct repository *r)
|
||||||
r->settings.pack_use_sparse = value;
|
r->settings.pack_use_sparse = value;
|
||||||
UPDATE_DEFAULT_BOOL(r->settings.pack_use_sparse, 1);
|
UPDATE_DEFAULT_BOOL(r->settings.pack_use_sparse, 1);
|
||||||
|
|
||||||
|
value = git_env_bool(GIT_TEST_MULTI_PACK_INDEX, 0);
|
||||||
|
if (value || !repo_config_get_bool(r, "core.multipackindex", &value))
|
||||||
|
r->settings.core_multi_pack_index = value;
|
||||||
|
UPDATE_DEFAULT_BOOL(r->settings.core_multi_pack_index, 1);
|
||||||
|
|
||||||
if (!repo_config_get_bool(r, "feature.manyfiles", &value) && value) {
|
if (!repo_config_get_bool(r, "feature.manyfiles", &value) && value) {
|
||||||
UPDATE_DEFAULT_BOOL(r->settings.index_version, 4);
|
UPDATE_DEFAULT_BOOL(r->settings.index_version, 4);
|
||||||
UPDATE_DEFAULT_BOOL(r->settings.core_untracked_cache, UNTRACKED_CACHE_WRITE);
|
UPDATE_DEFAULT_BOOL(r->settings.core_untracked_cache, UNTRACKED_CACHE_WRITE);
|
||||||
|
|
|
@ -39,6 +39,8 @@ struct repo_settings {
|
||||||
|
|
||||||
int pack_use_sparse;
|
int pack_use_sparse;
|
||||||
enum fetch_negotiation_setting fetch_negotiation_algorithm;
|
enum fetch_negotiation_setting fetch_negotiation_algorithm;
|
||||||
|
|
||||||
|
int core_multi_pack_index;
|
||||||
};
|
};
|
||||||
|
|
||||||
struct repository {
|
struct repository {
|
||||||
|
|
|
@ -3,6 +3,7 @@
|
||||||
test_description='multi-pack-indexes'
|
test_description='multi-pack-indexes'
|
||||||
. ./test-lib.sh
|
. ./test-lib.sh
|
||||||
|
|
||||||
|
GIT_TEST_MULTI_PACK_INDEX=0
|
||||||
objdir=.git/objects
|
objdir=.git/objects
|
||||||
|
|
||||||
HASH_LEN=$(test_oid rawsz)
|
HASH_LEN=$(test_oid rawsz)
|
||||||
|
@ -173,12 +174,12 @@ test_expect_success 'write progress off for redirected stderr' '
|
||||||
'
|
'
|
||||||
|
|
||||||
test_expect_success 'write force progress on for stderr' '
|
test_expect_success 'write force progress on for stderr' '
|
||||||
git multi-pack-index --object-dir=$objdir --progress write 2>err &&
|
GIT_PROGRESS_DELAY=0 git multi-pack-index --object-dir=$objdir --progress write 2>err &&
|
||||||
test_file_not_empty err
|
test_file_not_empty err
|
||||||
'
|
'
|
||||||
|
|
||||||
test_expect_success 'write with the --no-progress option' '
|
test_expect_success 'write with the --no-progress option' '
|
||||||
git multi-pack-index --object-dir=$objdir --no-progress write 2>err &&
|
GIT_PROGRESS_DELAY=0 git multi-pack-index --object-dir=$objdir --no-progress write 2>err &&
|
||||||
test_line_count = 0 err
|
test_line_count = 0 err
|
||||||
'
|
'
|
||||||
|
|
||||||
|
@ -368,17 +369,17 @@ test_expect_success 'git-fsck incorrect offset' '
|
||||||
'
|
'
|
||||||
|
|
||||||
test_expect_success 'repack progress off for redirected stderr' '
|
test_expect_success 'repack progress off for redirected stderr' '
|
||||||
git multi-pack-index --object-dir=$objdir repack 2>err &&
|
GIT_PROGRESS_DELAY=0 git multi-pack-index --object-dir=$objdir repack 2>err &&
|
||||||
test_line_count = 0 err
|
test_line_count = 0 err
|
||||||
'
|
'
|
||||||
|
|
||||||
test_expect_success 'repack force progress on for stderr' '
|
test_expect_success 'repack force progress on for stderr' '
|
||||||
git multi-pack-index --object-dir=$objdir --progress repack 2>err &&
|
GIT_PROGRESS_DELAY=0 git multi-pack-index --object-dir=$objdir --progress repack 2>err &&
|
||||||
test_file_not_empty err
|
test_file_not_empty err
|
||||||
'
|
'
|
||||||
|
|
||||||
test_expect_success 'repack with the --no-progress option' '
|
test_expect_success 'repack with the --no-progress option' '
|
||||||
git multi-pack-index --object-dir=$objdir --no-progress repack 2>err &&
|
GIT_PROGRESS_DELAY=0 git multi-pack-index --object-dir=$objdir --no-progress repack 2>err &&
|
||||||
test_line_count = 0 err
|
test_line_count = 0 err
|
||||||
'
|
'
|
||||||
|
|
||||||
|
@ -562,7 +563,7 @@ test_expect_success 'expire progress off for redirected stderr' '
|
||||||
test_expect_success 'expire force progress on for stderr' '
|
test_expect_success 'expire force progress on for stderr' '
|
||||||
(
|
(
|
||||||
cd dup &&
|
cd dup &&
|
||||||
git multi-pack-index --progress expire 2>err &&
|
GIT_PROGRESS_DELAY=0 git multi-pack-index --progress expire 2>err &&
|
||||||
test_file_not_empty err
|
test_file_not_empty err
|
||||||
)
|
)
|
||||||
'
|
'
|
||||||
|
@ -570,7 +571,7 @@ test_expect_success 'expire force progress on for stderr' '
|
||||||
test_expect_success 'expire with the --no-progress option' '
|
test_expect_success 'expire with the --no-progress option' '
|
||||||
(
|
(
|
||||||
cd dup &&
|
cd dup &&
|
||||||
git multi-pack-index --no-progress expire 2>err &&
|
GIT_PROGRESS_DELAY=0 git multi-pack-index --no-progress expire 2>err &&
|
||||||
test_line_count = 0 err
|
test_line_count = 0 err
|
||||||
)
|
)
|
||||||
'
|
'
|
||||||
|
|
|
@ -5,6 +5,7 @@ test_description='git maintenance builtin'
|
||||||
. ./test-lib.sh
|
. ./test-lib.sh
|
||||||
|
|
||||||
GIT_TEST_COMMIT_GRAPH=0
|
GIT_TEST_COMMIT_GRAPH=0
|
||||||
|
GIT_TEST_MULTI_PACK_INDEX=0
|
||||||
|
|
||||||
test_expect_success 'help text' '
|
test_expect_success 'help text' '
|
||||||
test_expect_code 129 git maintenance -h 2>err &&
|
test_expect_code 129 git maintenance -h 2>err &&
|
||||||
|
@ -62,4 +63,188 @@ test_expect_success 'run --task duplicate' '
|
||||||
test_i18ngrep "cannot be selected multiple times" err
|
test_i18ngrep "cannot be selected multiple times" err
|
||||||
'
|
'
|
||||||
|
|
||||||
|
test_expect_success 'run --task=prefetch with no remotes' '
|
||||||
|
git maintenance run --task=prefetch 2>err &&
|
||||||
|
test_must_be_empty err
|
||||||
|
'
|
||||||
|
|
||||||
|
test_expect_success 'prefetch multiple remotes' '
|
||||||
|
git clone . clone1 &&
|
||||||
|
git clone . clone2 &&
|
||||||
|
git remote add remote1 "file://$(pwd)/clone1" &&
|
||||||
|
git remote add remote2 "file://$(pwd)/clone2" &&
|
||||||
|
git -C clone1 switch -c one &&
|
||||||
|
git -C clone2 switch -c two &&
|
||||||
|
test_commit -C clone1 one &&
|
||||||
|
test_commit -C clone2 two &&
|
||||||
|
GIT_TRACE2_EVENT="$(pwd)/run-prefetch.txt" git maintenance run --task=prefetch 2>/dev/null &&
|
||||||
|
fetchargs="--prune --no-tags --no-write-fetch-head --recurse-submodules=no --refmap= --quiet" &&
|
||||||
|
test_subcommand git fetch remote1 $fetchargs +refs/heads/\\*:refs/prefetch/remote1/\\* <run-prefetch.txt &&
|
||||||
|
test_subcommand git fetch remote2 $fetchargs +refs/heads/\\*:refs/prefetch/remote2/\\* <run-prefetch.txt &&
|
||||||
|
test_path_is_missing .git/refs/remotes &&
|
||||||
|
git log prefetch/remote1/one &&
|
||||||
|
git log prefetch/remote2/two &&
|
||||||
|
git fetch --all &&
|
||||||
|
test_cmp_rev refs/remotes/remote1/one refs/prefetch/remote1/one &&
|
||||||
|
test_cmp_rev refs/remotes/remote2/two refs/prefetch/remote2/two
|
||||||
|
'
|
||||||
|
|
||||||
|
test_expect_success 'loose-objects task' '
|
||||||
|
# Repack everything so we know the state of the object dir
|
||||||
|
git repack -adk &&
|
||||||
|
|
||||||
|
# Hack to stop maintenance from running during "git commit"
|
||||||
|
echo in use >.git/objects/maintenance.lock &&
|
||||||
|
|
||||||
|
# Assuming that "git commit" creates at least one loose object
|
||||||
|
test_commit create-loose-object &&
|
||||||
|
rm .git/objects/maintenance.lock &&
|
||||||
|
|
||||||
|
ls .git/objects >obj-dir-before &&
|
||||||
|
test_file_not_empty obj-dir-before &&
|
||||||
|
ls .git/objects/pack/*.pack >packs-before &&
|
||||||
|
test_line_count = 1 packs-before &&
|
||||||
|
|
||||||
|
# The first run creates a pack-file
|
||||||
|
# but does not delete loose objects.
|
||||||
|
git maintenance run --task=loose-objects &&
|
||||||
|
ls .git/objects >obj-dir-between &&
|
||||||
|
test_cmp obj-dir-before obj-dir-between &&
|
||||||
|
ls .git/objects/pack/*.pack >packs-between &&
|
||||||
|
test_line_count = 2 packs-between &&
|
||||||
|
ls .git/objects/pack/loose-*.pack >loose-packs &&
|
||||||
|
test_line_count = 1 loose-packs &&
|
||||||
|
|
||||||
|
# The second run deletes loose objects
|
||||||
|
# but does not create a pack-file.
|
||||||
|
git maintenance run --task=loose-objects &&
|
||||||
|
ls .git/objects >obj-dir-after &&
|
||||||
|
cat >expect <<-\EOF &&
|
||||||
|
info
|
||||||
|
pack
|
||||||
|
EOF
|
||||||
|
test_cmp expect obj-dir-after &&
|
||||||
|
ls .git/objects/pack/*.pack >packs-after &&
|
||||||
|
test_cmp packs-between packs-after
|
||||||
|
'
|
||||||
|
|
||||||
|
test_expect_success 'maintenance.loose-objects.auto' '
|
||||||
|
git repack -adk &&
|
||||||
|
GIT_TRACE2_EVENT="$(pwd)/trace-lo1.txt" \
|
||||||
|
git -c maintenance.loose-objects.auto=1 maintenance \
|
||||||
|
run --auto --task=loose-objects 2>/dev/null &&
|
||||||
|
test_subcommand ! git prune-packed --quiet <trace-lo1.txt &&
|
||||||
|
printf data-A | git hash-object -t blob --stdin -w &&
|
||||||
|
GIT_TRACE2_EVENT="$(pwd)/trace-loA" \
|
||||||
|
git -c maintenance.loose-objects.auto=2 \
|
||||||
|
maintenance run --auto --task=loose-objects 2>/dev/null &&
|
||||||
|
test_subcommand ! git prune-packed --quiet <trace-loA &&
|
||||||
|
printf data-B | git hash-object -t blob --stdin -w &&
|
||||||
|
GIT_TRACE2_EVENT="$(pwd)/trace-loB" \
|
||||||
|
git -c maintenance.loose-objects.auto=2 \
|
||||||
|
maintenance run --auto --task=loose-objects 2>/dev/null &&
|
||||||
|
test_subcommand git prune-packed --quiet <trace-loB &&
|
||||||
|
GIT_TRACE2_EVENT="$(pwd)/trace-loC" \
|
||||||
|
git -c maintenance.loose-objects.auto=2 \
|
||||||
|
maintenance run --auto --task=loose-objects 2>/dev/null &&
|
||||||
|
test_subcommand git prune-packed --quiet <trace-loC
|
||||||
|
'
|
||||||
|
|
||||||
|
test_expect_success 'incremental-repack task' '
|
||||||
|
packDir=.git/objects/pack &&
|
||||||
|
for i in $(test_seq 1 5)
|
||||||
|
do
|
||||||
|
test_commit $i || return 1
|
||||||
|
done &&
|
||||||
|
|
||||||
|
# Create three disjoint pack-files with size BIG, small, small.
|
||||||
|
echo HEAD~2 | git pack-objects --revs $packDir/test-1 &&
|
||||||
|
test_tick &&
|
||||||
|
git pack-objects --revs $packDir/test-2 <<-\EOF &&
|
||||||
|
HEAD~1
|
||||||
|
^HEAD~2
|
||||||
|
EOF
|
||||||
|
test_tick &&
|
||||||
|
git pack-objects --revs $packDir/test-3 <<-\EOF &&
|
||||||
|
HEAD
|
||||||
|
^HEAD~1
|
||||||
|
EOF
|
||||||
|
rm -f $packDir/pack-* &&
|
||||||
|
rm -f $packDir/loose-* &&
|
||||||
|
ls $packDir/*.pack >packs-before &&
|
||||||
|
test_line_count = 3 packs-before &&
|
||||||
|
|
||||||
|
# the job repacks the two into a new pack, but does not
|
||||||
|
# delete the old ones.
|
||||||
|
git maintenance run --task=incremental-repack &&
|
||||||
|
ls $packDir/*.pack >packs-between &&
|
||||||
|
test_line_count = 4 packs-between &&
|
||||||
|
|
||||||
|
# the job deletes the two old packs, and does not write
|
||||||
|
# a new one because the batch size is not high enough to
|
||||||
|
# pack the largest pack-file.
|
||||||
|
git maintenance run --task=incremental-repack &&
|
||||||
|
ls .git/objects/pack/*.pack >packs-after &&
|
||||||
|
test_line_count = 2 packs-after
|
||||||
|
'
|
||||||
|
|
||||||
|
test_expect_success EXPENSIVE 'incremental-repack 2g limit' '
|
||||||
|
for i in $(test_seq 1 5)
|
||||||
|
do
|
||||||
|
test-tool genrandom foo$i $((512 * 1024 * 1024 + 1)) >>big ||
|
||||||
|
return 1
|
||||||
|
done &&
|
||||||
|
git add big &&
|
||||||
|
git commit -m "Add big file (1)" &&
|
||||||
|
|
||||||
|
# ensure any possible loose objects are in a pack-file
|
||||||
|
git maintenance run --task=loose-objects &&
|
||||||
|
|
||||||
|
rm big &&
|
||||||
|
for i in $(test_seq 6 10)
|
||||||
|
do
|
||||||
|
test-tool genrandom foo$i $((512 * 1024 * 1024 + 1)) >>big ||
|
||||||
|
return 1
|
||||||
|
done &&
|
||||||
|
git add big &&
|
||||||
|
git commit -m "Add big file (2)" &&
|
||||||
|
|
||||||
|
# ensure any possible loose objects are in a pack-file
|
||||||
|
git maintenance run --task=loose-objects &&
|
||||||
|
|
||||||
|
# Now run the incremental-repack task and check the batch-size
|
||||||
|
GIT_TRACE2_EVENT="$(pwd)/run-2g.txt" git maintenance run \
|
||||||
|
--task=incremental-repack 2>/dev/null &&
|
||||||
|
test_subcommand git multi-pack-index repack \
|
||||||
|
--no-progress --batch-size=2147483647 <run-2g.txt
|
||||||
|
'
|
||||||
|
|
||||||
|
test_expect_success 'maintenance.incremental-repack.auto' '
|
||||||
|
git repack -adk &&
|
||||||
|
git config core.multiPackIndex true &&
|
||||||
|
git multi-pack-index write &&
|
||||||
|
GIT_TRACE2_EVENT="$(pwd)/midx-init.txt" git \
|
||||||
|
-c maintenance.incremental-repack.auto=1 \
|
||||||
|
maintenance run --auto --task=incremental-repack 2>/dev/null &&
|
||||||
|
test_subcommand ! git multi-pack-index write --no-progress <midx-init.txt &&
|
||||||
|
test_commit A &&
|
||||||
|
git pack-objects --revs .git/objects/pack/pack <<-\EOF &&
|
||||||
|
HEAD
|
||||||
|
^HEAD~1
|
||||||
|
EOF
|
||||||
|
GIT_TRACE2_EVENT=$(pwd)/trace-A git \
|
||||||
|
-c maintenance.incremental-repack.auto=2 \
|
||||||
|
maintenance run --auto --task=incremental-repack 2>/dev/null &&
|
||||||
|
test_subcommand ! git multi-pack-index write --no-progress <trace-A &&
|
||||||
|
test_commit B &&
|
||||||
|
git pack-objects --revs .git/objects/pack/pack <<-\EOF &&
|
||||||
|
HEAD
|
||||||
|
^HEAD~1
|
||||||
|
EOF
|
||||||
|
GIT_TRACE2_EVENT=$(pwd)/trace-B git \
|
||||||
|
-c maintenance.incremental-repack.auto=2 \
|
||||||
|
maintenance run --auto --task=incremental-repack 2>/dev/null &&
|
||||||
|
test_subcommand git multi-pack-index write --no-progress <trace-B
|
||||||
|
'
|
||||||
|
|
||||||
test_done
|
test_done
|
||||||
|
|
Loading…
Reference in New Issue