mirror of https://github.com/git/git.git
Merge branch 'jt/cdn-offload'
The "fetch/clone" protocol has been updated to allow the server to instruct the clients to grab pre-packaged packfile(s) in addition to the packed object data coming over the wire. * jt/cdn-offload: upload-pack: fix a sparse '0 as NULL pointer' warning upload-pack: send part of packfile response as uri fetch-pack: support more than one pack lockfile upload-pack: refactor reading of pack-objects out Documentation: add Packfile URIs design doc Documentation: order protocol v2 sections http-fetch: support fetching packfiles by URL http-fetch: refactor into function http: refactor finish_http_pack_request() http: use --stdin when indexing dumb HTTP pack
This commit is contained in:
Junio C Hamano
commit
34e849b05a
|
|
@ -9,7 +9,7 @@ git-http-fetch - Download from a remote Git repository via HTTP
|
|||
SYNOPSIS
|
||||
--------
|
||||
[verse]
|
||||
'git http-fetch' [-c] [-t] [-a] [-d] [-v] [-w filename] [--recover] [--stdin] <commit> <url>
|
||||
'git http-fetch' [-c] [-t] [-a] [-d] [-v] [-w filename] [--recover] [--stdin | --packfile=<hash> | <commit>] <url>
|
||||
|
||||
DESCRIPTION
|
||||
-----------
|
||||
|
|
@ -40,6 +40,13 @@ commit-id::
|
|||
|
||||
<commit-id>['\t'<filename-as-in--w>]
|
||||
|
||||
--packfile=<hash>::
|
||||
Instead of a commit id on the command line (which is not expected in
|
||||
this case), 'git http-fetch' fetches the packfile directly at the given
|
||||
URL and uses index-pack to generate corresponding .idx and .keep files.
|
||||
The hash is used to determine the name of the temporary file and is
|
||||
arbitrary. The output of index-pack is printed to stdout.
|
||||
|
||||
--recover::
|
||||
Verify that everything reachable from target is fetched. Used after
|
||||
an earlier fetch is interrupted.
|
||||
|
|
|
|||
|
|
@ -0,0 +1,78 @@
|
|||
Packfile URIs
|
||||
=============
|
||||
|
||||
This feature allows servers to serve part of their packfile response as URIs.
|
||||
This allows server designs that improve scalability in bandwidth and CPU usage
|
||||
(for example, by serving some data through a CDN), and (in the future) provides
|
||||
some measure of resumability to clients.
|
||||
|
||||
This feature is available only in protocol version 2.
|
||||
|
||||
Protocol
|
||||
--------
|
||||
|
||||
The server advertises the `packfile-uris` capability.
|
||||
|
||||
If the client then communicates which protocols (HTTPS, etc.) it supports with
|
||||
a `packfile-uris` argument, the server MAY send a `packfile-uris` section
|
||||
directly before the `packfile` section (right after `wanted-refs` if it is
|
||||
sent) containing URIs of any of the given protocols. The URIs point to
|
||||
packfiles that use only features that the client has declared that it supports
|
||||
(e.g. ofs-delta and thin-pack). See protocol-v2.txt for the documentation of
|
||||
this section.
|
||||
|
||||
Clients should then download and index all the given URIs (in addition to
|
||||
downloading and indexing the packfile given in the `packfile` section of the
|
||||
response) before performing the connectivity check.
|
||||
|
||||
Server design
|
||||
-------------
|
||||
|
||||
The server can be trivially made compatible with the proposed protocol by
|
||||
having it advertise `packfile-uris`, tolerating the client sending
|
||||
`packfile-uris`, and never sending any `packfile-uris` section. But we should
|
||||
include some sort of non-trivial implementation in the Minimum Viable Product,
|
||||
at least so that we can test the client.
|
||||
|
||||
This is the implementation: a feature, marked experimental, that allows the
|
||||
server to be configured by one or more `uploadpack.blobPackfileUri=<sha1>
|
||||
<uri>` entries. Whenever the list of objects to be sent is assembled, all such
|
||||
blobs are excluded, replaced with URIs. The client will download those URIs,
|
||||
expecting them to each point to packfiles containing single blobs.
|
||||
|
||||
Client design
|
||||
-------------
|
||||
|
||||
The client has a config variable `fetch.uriprotocols` that determines which
|
||||
protocols the end user is willing to use. By default, this is empty.
|
||||
|
||||
When the client downloads the given URIs, it should store them with "keep"
|
||||
files, just like it does with the packfile in the `packfile` section. These
|
||||
additional "keep" files can only be removed after the refs have been updated -
|
||||
just like the "keep" file for the packfile in the `packfile` section.
|
||||
|
||||
The division of work (initial fetch + additional URIs) introduces convenient
|
||||
points for resumption of an interrupted clone - such resumption can be done
|
||||
after the Minimum Viable Product (see "Future work").
|
||||
|
||||
Future work
|
||||
-----------
|
||||
|
||||
The protocol design allows some evolution of the server and client without any
|
||||
need for protocol changes, so only a small-scoped design is included here to
|
||||
form the MVP. For example, the following can be done:
|
||||
|
||||
* On the server, more sophisticated means of excluding objects (e.g. by
|
||||
specifying a commit to represent that commit and all objects that it
|
||||
references).
|
||||
* On the client, resumption of clone. If a clone is interrupted, information
|
||||
could be recorded in the repository's config and a "clone-resume" command
|
||||
can resume the clone in progress. (Resumption of subsequent fetches is more
|
||||
difficult because that must deal with the user wanting to use the repository
|
||||
even after the fetch was interrupted.)
|
||||
|
||||
There are some possible features that will require a change in protocol:
|
||||
|
||||
* Additional HTTP headers (e.g. authentication)
|
||||
* Byte range support
|
||||
* Different file formats referenced by URIs (e.g. raw object)
|
||||
|
|
@ -325,13 +325,26 @@ included in the client's request:
|
|||
indicating its sideband (1, 2, or 3), and the server may send "0005\2"
|
||||
(a PKT-LINE of sideband 2 with no payload) as a keepalive packet.
|
||||
|
||||
If the 'packfile-uris' feature is advertised, the following argument
|
||||
can be included in the client's request as well as the potential
|
||||
addition of the 'packfile-uris' section in the server's response as
|
||||
explained below.
|
||||
|
||||
packfile-uris <comma-separated list of protocols>
|
||||
Indicates to the server that the client is willing to receive
|
||||
URIs of any of the given protocols in place of objects in the
|
||||
sent packfile. Before performing the connectivity check, the
|
||||
client should download from all given URIs. Currently, the
|
||||
protocols supported are "http" and "https".
|
||||
|
||||
The response of `fetch` is broken into a number of sections separated by
|
||||
delimiter packets (0001), with each section beginning with its section
|
||||
header.
|
||||
header. Most sections are sent only when the packfile is sent.
|
||||
|
||||
output = *section
|
||||
section = (acknowledgments | shallow-info | wanted-refs | packfile)
|
||||
(flush-pkt | delim-pkt)
|
||||
output = acknowledgements flush-pkt |
|
||||
[acknowledgments delim-pkt] [shallow-info delim-pkt]
|
||||
[wanted-refs delim-pkt] [packfile-uris delim-pkt]
|
||||
packfile flush-pkt
|
||||
|
||||
acknowledgments = PKT-LINE("acknowledgments" LF)
|
||||
(nak | *ack)
|
||||
|
|
@ -349,13 +362,17 @@ header.
|
|||
*PKT-LINE(wanted-ref LF)
|
||||
wanted-ref = obj-id SP refname
|
||||
|
||||
packfile-uris = PKT-LINE("packfile-uris" LF) *packfile-uri
|
||||
packfile-uri = PKT-LINE(40*(HEXDIGIT) SP *%x20-ff LF)
|
||||
|
||||
packfile = PKT-LINE("packfile" LF)
|
||||
*PKT-LINE(%x01-03 *%x00-ff)
|
||||
|
||||
acknowledgments section
|
||||
* If the client determines that it is finished with negotiations
|
||||
by sending a "done" line, the acknowledgments sections MUST be
|
||||
omitted from the server's response.
|
||||
* If the client determines that it is finished with negotiations by
|
||||
sending a "done" line (thus requiring the server to send a packfile),
|
||||
the acknowledgments sections MUST be omitted from the server's
|
||||
response.
|
||||
|
||||
* Always begins with the section header "acknowledgments"
|
||||
|
||||
|
|
@ -406,9 +423,6 @@ header.
|
|||
which the client has not indicated was shallow as a part of
|
||||
its request.
|
||||
|
||||
* This section is only included if a packfile section is also
|
||||
included in the response.
|
||||
|
||||
wanted-refs section
|
||||
* This section is only included if the client has requested a
|
||||
ref using a 'want-ref' line and if a packfile section is also
|
||||
|
|
@ -422,6 +436,20 @@ header.
|
|||
* The server MUST NOT send any refs which were not requested
|
||||
using 'want-ref' lines.
|
||||
|
||||
packfile-uris section
|
||||
* This section is only included if the client sent
|
||||
'packfile-uris' and the server has at least one such URI to
|
||||
send.
|
||||
|
||||
* Always begins with the section header "packfile-uris".
|
||||
|
||||
* For each URI the server sends, it sends a hash of the pack's
|
||||
contents (as output by git index-pack) followed by the URI.
|
||||
|
||||
* The hashes are 40 hex characters long. When Git upgrades to a new
|
||||
hash algorithm, this might need to be updated. (It should match
|
||||
whatever index-pack outputs after "pack\t" or "keep\t".
|
||||
|
||||
packfile section
|
||||
* This section is only included if the client has sent 'want'
|
||||
lines in its request and either requested that no more
|
||||
|
|
|
|||
|
|
@ -48,8 +48,8 @@ int cmd_fetch_pack(int argc, const char **argv, const char *prefix)
|
|||
struct ref **sought = NULL;
|
||||
int nr_sought = 0, alloc_sought = 0;
|
||||
int fd[2];
|
||||
char *pack_lockfile = NULL;
|
||||
char **pack_lockfile_ptr = NULL;
|
||||
struct string_list pack_lockfiles = STRING_LIST_INIT_DUP;
|
||||
struct string_list *pack_lockfiles_ptr = NULL;
|
||||
struct child_process *conn;
|
||||
struct fetch_pack_args args;
|
||||
struct oid_array shallow = OID_ARRAY_INIT;
|
||||
|
|
@ -134,7 +134,7 @@ int cmd_fetch_pack(int argc, const char **argv, const char *prefix)
|
|||
}
|
||||
if (!strcmp("--lock-pack", arg)) {
|
||||
args.lock_pack = 1;
|
||||
pack_lockfile_ptr = &pack_lockfile;
|
||||
pack_lockfiles_ptr = &pack_lockfiles;
|
||||
continue;
|
||||
}
|
||||
if (!strcmp("--check-self-contained-and-connected", arg)) {
|
||||
|
|
@ -235,10 +235,15 @@ int cmd_fetch_pack(int argc, const char **argv, const char *prefix)
|
|||
}
|
||||
|
||||
ref = fetch_pack(&args, fd, ref, sought, nr_sought,
|
||||
&shallow, pack_lockfile_ptr, version);
|
||||
if (pack_lockfile) {
|
||||
printf("lock %s\n", pack_lockfile);
|
||||
&shallow, pack_lockfiles_ptr, version);
|
||||
if (pack_lockfiles.nr) {
|
||||
int i;
|
||||
|
||||
printf("lock %s\n", pack_lockfiles.items[0].string);
|
||||
fflush(stdout);
|
||||
for (i = 1; i < pack_lockfiles.nr; i++)
|
||||
warning(_("Lockfile created but not reported: %s"),
|
||||
pack_lockfiles.items[i].string);
|
||||
}
|
||||
if (args.check_self_contained_and_connected &&
|
||||
args.self_contained_and_connected) {
|
||||
|
|
|
|||
|
|
@ -117,6 +117,8 @@ static unsigned long window_memory_limit = 0;
|
|||
|
||||
static struct list_objects_filter_options filter_options;
|
||||
|
||||
static struct string_list uri_protocols = STRING_LIST_INIT_NODUP;
|
||||
|
||||
enum missing_action {
|
||||
MA_ERROR = 0, /* fail if any missing objects are encountered */
|
||||
MA_ALLOW_ANY, /* silently allow ALL missing objects */
|
||||
|
|
@ -125,6 +127,15 @@ enum missing_action {
|
|||
static enum missing_action arg_missing_action;
|
||||
static show_object_fn fn_show_object;
|
||||
|
||||
struct configured_exclusion {
|
||||
struct oidmap_entry e;
|
||||
char *pack_hash_hex;
|
||||
char *uri;
|
||||
};
|
||||
static struct oidmap configured_exclusions;
|
||||
|
||||
static struct oidset excluded_by_config;
|
||||
|
||||
/*
|
||||
* stats
|
||||
*/
|
||||
|
|
@ -969,6 +980,25 @@ static void write_reused_pack(struct hashfile *f)
|
|||
unuse_pack(&w_curs);
|
||||
}
|
||||
|
||||
static void write_excluded_by_configs(void)
|
||||
{
|
||||
struct oidset_iter iter;
|
||||
const struct object_id *oid;
|
||||
|
||||
oidset_iter_init(&excluded_by_config, &iter);
|
||||
while ((oid = oidset_iter_next(&iter))) {
|
||||
struct configured_exclusion *ex =
|
||||
oidmap_get(&configured_exclusions, oid);
|
||||
|
||||
if (!ex)
|
||||
BUG("configured exclusion wasn't configured");
|
||||
write_in_full(1, ex->pack_hash_hex, strlen(ex->pack_hash_hex));
|
||||
write_in_full(1, " ", 1);
|
||||
write_in_full(1, ex->uri, strlen(ex->uri));
|
||||
write_in_full(1, "\n", 1);
|
||||
}
|
||||
}
|
||||
|
||||
static const char no_split_warning[] = N_(
|
||||
"disabling bitmap writing, packs are split due to pack.packSizeLimit"
|
||||
);
|
||||
|
|
@ -1266,6 +1296,25 @@ static int want_object_in_pack(const struct object_id *oid,
|
|||
}
|
||||
}
|
||||
|
||||
if (uri_protocols.nr) {
|
||||
struct configured_exclusion *ex =
|
||||
oidmap_get(&configured_exclusions, oid);
|
||||
int i;
|
||||
const char *p;
|
||||
|
||||
if (ex) {
|
||||
for (i = 0; i < uri_protocols.nr; i++) {
|
||||
if (skip_prefix(ex->uri,
|
||||
uri_protocols.items[i].string,
|
||||
&p) &&
|
||||
*p == ':') {
|
||||
oidset_insert(&excluded_by_config, oid);
|
||||
return 0;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return 1;
|
||||
}
|
||||
|
||||
|
|
@ -2864,6 +2913,29 @@ static int git_pack_config(const char *k, const char *v, void *cb)
|
|||
pack_idx_opts.version);
|
||||
return 0;
|
||||
}
|
||||
if (!strcmp(k, "uploadpack.blobpackfileuri")) {
|
||||
struct configured_exclusion *ex = xmalloc(sizeof(*ex));
|
||||
const char *oid_end, *pack_end;
|
||||
/*
|
||||
* Stores the pack hash. This is not a true object ID, but is
|
||||
* of the same form.
|
||||
*/
|
||||
struct object_id pack_hash;
|
||||
|
||||
if (parse_oid_hex(v, &ex->e.oid, &oid_end) ||
|
||||
*oid_end != ' ' ||
|
||||
parse_oid_hex(oid_end + 1, &pack_hash, &pack_end) ||
|
||||
*pack_end != ' ')
|
||||
die(_("value of uploadpack.blobpackfileuri must be "
|
||||
"of the form '<object-hash> <pack-hash> <uri>' (got '%s')"), v);
|
||||
if (oidmap_get(&configured_exclusions, &ex->e.oid))
|
||||
die(_("object already configured in another "
|
||||
"uploadpack.blobpackfileuri (got '%s')"), v);
|
||||
ex->pack_hash_hex = xcalloc(1, pack_end - oid_end);
|
||||
memcpy(ex->pack_hash_hex, oid_end + 1, pack_end - oid_end - 1);
|
||||
ex->uri = xstrdup(pack_end + 1);
|
||||
oidmap_put(&configured_exclusions, ex);
|
||||
}
|
||||
return git_default_config(k, v, cb);
|
||||
}
|
||||
|
||||
|
|
@ -3462,6 +3534,9 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)
|
|||
N_("do not pack objects in promisor packfiles")),
|
||||
OPT_BOOL(0, "delta-islands", &use_delta_islands,
|
||||
N_("respect islands during delta compression")),
|
||||
OPT_STRING_LIST(0, "uri-protocol", &uri_protocols,
|
||||
N_("protocol"),
|
||||
N_("exclude any configured uploadpack.blobpackfileuri with this protocol")),
|
||||
OPT_END(),
|
||||
};
|
||||
|
||||
|
|
@ -3650,6 +3725,7 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)
|
|||
}
|
||||
|
||||
trace2_region_enter("pack-objects", "write-pack-file", the_repository);
|
||||
write_excluded_by_configs();
|
||||
write_pack_file();
|
||||
trace2_region_leave("pack-objects", "write-pack-file", the_repository);
|
||||
|
||||
|
|
|
|||
|
|
@ -43,10 +43,12 @@ int check_connected(oid_iterate_fn fn, void *cb_data,
|
|||
|
||||
if (transport && transport->smart_options &&
|
||||
transport->smart_options->self_contained_and_connected &&
|
||||
transport->pack_lockfile &&
|
||||
strip_suffix(transport->pack_lockfile, ".keep", &base_len)) {
|
||||
transport->pack_lockfiles.nr == 1 &&
|
||||
strip_suffix(transport->pack_lockfiles.items[0].string,
|
||||
".keep", &base_len)) {
|
||||
struct strbuf idx_file = STRBUF_INIT;
|
||||
strbuf_add(&idx_file, transport->pack_lockfile, base_len);
|
||||
strbuf_add(&idx_file, transport->pack_lockfiles.items[0].string,
|
||||
base_len);
|
||||
strbuf_addstr(&idx_file, ".idx");
|
||||
new_pack = add_packed_git(idx_file.buf, idx_file.len, 1);
|
||||
strbuf_release(&idx_file);
|
||||
|
|
|
|||
137
fetch-pack.c
137
fetch-pack.c
|
|
@ -38,6 +38,7 @@ static int server_supports_filtering;
|
|||
static struct shallow_lock shallow_lock;
|
||||
static const char *alternate_shallow_file;
|
||||
static struct strbuf fsck_msg_types = STRBUF_INIT;
|
||||
static struct string_list uri_protocols = STRING_LIST_INIT_DUP;
|
||||
|
||||
/* Remember to update object flag allocation in object.h */
|
||||
#define COMPLETE (1U << 0)
|
||||
|
|
@ -794,7 +795,8 @@ static void write_promisor_file(const char *keep_name,
|
|||
}
|
||||
|
||||
static int get_pack(struct fetch_pack_args *args,
|
||||
int xd[2], char **pack_lockfile,
|
||||
int xd[2], struct string_list *pack_lockfiles,
|
||||
int only_packfile,
|
||||
struct ref **sought, int nr_sought)
|
||||
{
|
||||
struct async demux;
|
||||
|
|
@ -838,7 +840,7 @@ static int get_pack(struct fetch_pack_args *args,
|
|||
}
|
||||
|
||||
if (do_keep || args->from_promisor) {
|
||||
if (pack_lockfile)
|
||||
if (pack_lockfiles)
|
||||
cmd.out = -1;
|
||||
cmd_name = "index-pack";
|
||||
argv_array_push(&cmd.args, cmd_name);
|
||||
|
|
@ -855,15 +857,22 @@ static int get_pack(struct fetch_pack_args *args,
|
|||
"--keep=fetch-pack %"PRIuMAX " on %s",
|
||||
(uintmax_t)getpid(), hostname);
|
||||
}
|
||||
if (args->check_self_contained_and_connected)
|
||||
if (only_packfile && args->check_self_contained_and_connected)
|
||||
argv_array_push(&cmd.args, "--check-self-contained-and-connected");
|
||||
else
|
||||
/*
|
||||
* We cannot perform any connectivity checks because
|
||||
* not all packs have been downloaded; let the caller
|
||||
* have this responsibility.
|
||||
*/
|
||||
args->check_self_contained_and_connected = 0;
|
||||
/*
|
||||
* If we're obtaining the filename of a lockfile, we'll use
|
||||
* that filename to write a .promisor file with more
|
||||
* information below. If not, we need index-pack to do it for
|
||||
* us.
|
||||
*/
|
||||
if (!(do_keep && pack_lockfile) && args->from_promisor)
|
||||
if (!(do_keep && pack_lockfiles) && args->from_promisor)
|
||||
argv_array_push(&cmd.args, "--promisor");
|
||||
}
|
||||
else {
|
||||
|
|
@ -899,8 +908,9 @@ static int get_pack(struct fetch_pack_args *args,
|
|||
cmd.git_cmd = 1;
|
||||
if (start_command(&cmd))
|
||||
die(_("fetch-pack: unable to fork off %s"), cmd_name);
|
||||
if (do_keep && pack_lockfile) {
|
||||
*pack_lockfile = index_pack_lockfile(cmd.out);
|
||||
if (do_keep && pack_lockfiles) {
|
||||
string_list_append_nodup(pack_lockfiles,
|
||||
index_pack_lockfile(cmd.out));
|
||||
close(cmd.out);
|
||||
}
|
||||
|
||||
|
|
@ -922,8 +932,8 @@ static int get_pack(struct fetch_pack_args *args,
|
|||
* Now that index-pack has succeeded, write the promisor file using the
|
||||
* obtained .keep filename if necessary
|
||||
*/
|
||||
if (do_keep && pack_lockfile && args->from_promisor)
|
||||
write_promisor_file(*pack_lockfile, sought, nr_sought);
|
||||
if (do_keep && pack_lockfiles && pack_lockfiles->nr && args->from_promisor)
|
||||
write_promisor_file(pack_lockfiles->items[0].string, sought, nr_sought);
|
||||
|
||||
return 0;
|
||||
}
|
||||
|
|
@ -940,7 +950,7 @@ static struct ref *do_fetch_pack(struct fetch_pack_args *args,
|
|||
const struct ref *orig_ref,
|
||||
struct ref **sought, int nr_sought,
|
||||
struct shallow_info *si,
|
||||
char **pack_lockfile)
|
||||
struct string_list *pack_lockfiles)
|
||||
{
|
||||
struct repository *r = the_repository;
|
||||
struct ref *ref = copy_ref_list(orig_ref);
|
||||
|
|
@ -1067,7 +1077,7 @@ static struct ref *do_fetch_pack(struct fetch_pack_args *args,
|
|||
alternate_shallow_file = setup_temporary_shallow(si->shallow);
|
||||
else
|
||||
alternate_shallow_file = NULL;
|
||||
if (get_pack(args, fd, pack_lockfile, sought, nr_sought))
|
||||
if (get_pack(args, fd, pack_lockfiles, 1, sought, nr_sought))
|
||||
die(_("git fetch-pack: fetch failed."));
|
||||
|
||||
all_done:
|
||||
|
|
@ -1221,6 +1231,26 @@ static int send_fetch_request(struct fetch_negotiator *negotiator, int fd_out,
|
|||
warning("filtering not recognized by server, ignoring");
|
||||
}
|
||||
|
||||
if (server_supports_feature("fetch", "packfile-uris", 0)) {
|
||||
int i;
|
||||
struct strbuf to_send = STRBUF_INIT;
|
||||
|
||||
for (i = 0; i < uri_protocols.nr; i++) {
|
||||
const char *s = uri_protocols.items[i].string;
|
||||
|
||||
if (!strcmp(s, "https") || !strcmp(s, "http")) {
|
||||
if (to_send.len)
|
||||
strbuf_addch(&to_send, ',');
|
||||
strbuf_addstr(&to_send, s);
|
||||
}
|
||||
}
|
||||
if (to_send.len) {
|
||||
packet_buf_write(&req_buf, "packfile-uris %s",
|
||||
to_send.buf);
|
||||
strbuf_release(&to_send);
|
||||
}
|
||||
}
|
||||
|
||||
/* add wants */
|
||||
add_wants(args->no_dependents, wants, &req_buf);
|
||||
|
||||
|
|
@ -1443,6 +1473,21 @@ static void receive_wanted_refs(struct packet_reader *reader,
|
|||
die(_("error processing wanted refs: %d"), reader->status);
|
||||
}
|
||||
|
||||
static void receive_packfile_uris(struct packet_reader *reader,
|
||||
struct string_list *uris)
|
||||
{
|
||||
process_section_header(reader, "packfile-uris", 0);
|
||||
while (packet_reader_read(reader) == PACKET_READ_NORMAL) {
|
||||
if (reader->pktlen < the_hash_algo->hexsz ||
|
||||
reader->line[the_hash_algo->hexsz] != ' ')
|
||||
die("expected '<hash> <uri>', got: %s\n", reader->line);
|
||||
|
||||
string_list_append(uris, reader->line);
|
||||
}
|
||||
if (reader->status != PACKET_READ_DELIM)
|
||||
die("expected DELIM");
|
||||
}
|
||||
|
||||
enum fetch_state {
|
||||
FETCH_CHECK_LOCAL = 0,
|
||||
FETCH_SEND_REQUEST,
|
||||
|
|
@ -1464,7 +1509,7 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,
|
|||
struct ref **sought, int nr_sought,
|
||||
struct oid_array *shallows,
|
||||
struct shallow_info *si,
|
||||
char **pack_lockfile)
|
||||
struct string_list *pack_lockfiles)
|
||||
{
|
||||
struct repository *r = the_repository;
|
||||
struct ref *ref = copy_ref_list(orig_ref);
|
||||
|
|
@ -1476,6 +1521,8 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,
|
|||
struct fetch_negotiator negotiator_alloc;
|
||||
struct fetch_negotiator *negotiator;
|
||||
int seen_ack = 0;
|
||||
struct string_list packfile_uris = STRING_LIST_INIT_DUP;
|
||||
int i;
|
||||
|
||||
if (args->no_dependents) {
|
||||
negotiator = NULL;
|
||||
|
|
@ -1569,9 +1616,12 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,
|
|||
if (process_section_header(&reader, "wanted-refs", 1))
|
||||
receive_wanted_refs(&reader, sought, nr_sought);
|
||||
|
||||
/* get the pack */
|
||||
/* get the pack(s) */
|
||||
if (process_section_header(&reader, "packfile-uris", 1))
|
||||
receive_packfile_uris(&reader, &packfile_uris);
|
||||
process_section_header(&reader, "packfile", 0);
|
||||
if (get_pack(args, fd, pack_lockfile, sought, nr_sought))
|
||||
if (get_pack(args, fd, pack_lockfiles,
|
||||
!packfile_uris.nr, sought, nr_sought))
|
||||
die(_("git fetch-pack: fetch failed."));
|
||||
do_check_stateless_delimiter(args, &reader);
|
||||
|
||||
|
|
@ -1582,8 +1632,55 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,
|
|||
}
|
||||
}
|
||||
|
||||
for (i = 0; i < packfile_uris.nr; i++) {
|
||||
struct child_process cmd = CHILD_PROCESS_INIT;
|
||||
char packname[GIT_MAX_HEXSZ + 1];
|
||||
const char *uri = packfile_uris.items[i].string +
|
||||
the_hash_algo->hexsz + 1;
|
||||
|
||||
argv_array_push(&cmd.args, "http-fetch");
|
||||
argv_array_pushf(&cmd.args, "--packfile=%.*s",
|
||||
(int) the_hash_algo->hexsz,
|
||||
packfile_uris.items[i].string);
|
||||
argv_array_push(&cmd.args, uri);
|
||||
cmd.git_cmd = 1;
|
||||
cmd.no_stdin = 1;
|
||||
cmd.out = -1;
|
||||
if (start_command(&cmd))
|
||||
die("fetch-pack: unable to spawn http-fetch");
|
||||
|
||||
if (read_in_full(cmd.out, packname, 5) < 0 ||
|
||||
memcmp(packname, "keep\t", 5))
|
||||
die("fetch-pack: expected keep then TAB at start of http-fetch output");
|
||||
|
||||
if (read_in_full(cmd.out, packname,
|
||||
the_hash_algo->hexsz + 1) < 0 ||
|
||||
packname[the_hash_algo->hexsz] != '\n')
|
||||
die("fetch-pack: expected hash then LF at end of http-fetch output");
|
||||
|
||||
packname[the_hash_algo->hexsz] = '\0';
|
||||
|
||||
close(cmd.out);
|
||||
|
||||
if (finish_command(&cmd))
|
||||
die("fetch-pack: unable to finish http-fetch");
|
||||
|
||||
if (memcmp(packfile_uris.items[i].string, packname,
|
||||
the_hash_algo->hexsz))
|
||||
die("fetch-pack: pack downloaded from %s does not match expected hash %.*s",
|
||||
uri, (int) the_hash_algo->hexsz,
|
||||
packfile_uris.items[i].string);
|
||||
|
||||
string_list_append_nodup(pack_lockfiles,
|
||||
xstrfmt("%s/pack/pack-%s.keep",
|
||||
get_object_directory(),
|
||||
packname));
|
||||
}
|
||||
string_list_clear(&packfile_uris, 0);
|
||||
|
||||
if (negotiator)
|
||||
negotiator->release(negotiator);
|
||||
|
||||
oidset_clear(&common);
|
||||
return ref;
|
||||
}
|
||||
|
|
@ -1620,6 +1717,14 @@ static void fetch_pack_config(void)
|
|||
git_config_get_bool("repack.usedeltabaseoffset", &prefer_ofs_delta);
|
||||
git_config_get_bool("fetch.fsckobjects", &fetch_fsck_objects);
|
||||
git_config_get_bool("transfer.fsckobjects", &transfer_fsck_objects);
|
||||
if (!uri_protocols.nr) {
|
||||
char *str;
|
||||
|
||||
if (!git_config_get_string("fetch.uriprotocols", &str) && str) {
|
||||
string_list_split(&uri_protocols, str, ',', -1);
|
||||
free(str);
|
||||
}
|
||||
}
|
||||
|
||||
git_config(fetch_pack_config_cb, NULL);
|
||||
}
|
||||
|
|
@ -1772,7 +1877,7 @@ struct ref *fetch_pack(struct fetch_pack_args *args,
|
|||
const struct ref *ref,
|
||||
struct ref **sought, int nr_sought,
|
||||
struct oid_array *shallow,
|
||||
char **pack_lockfile,
|
||||
struct string_list *pack_lockfiles,
|
||||
enum protocol_version version)
|
||||
{
|
||||
struct ref *ref_cpy;
|
||||
|
|
@ -1807,11 +1912,11 @@ struct ref *fetch_pack(struct fetch_pack_args *args,
|
|||
memset(&si, 0, sizeof(si));
|
||||
ref_cpy = do_fetch_pack_v2(args, fd, ref, sought, nr_sought,
|
||||
&shallows_scratch, &si,
|
||||
pack_lockfile);
|
||||
pack_lockfiles);
|
||||
} else {
|
||||
prepare_shallow_info(&si, shallow);
|
||||
ref_cpy = do_fetch_pack(args, fd, ref, sought, nr_sought,
|
||||
&si, pack_lockfile);
|
||||
&si, pack_lockfiles);
|
||||
}
|
||||
reprepare_packed_git(the_repository);
|
||||
|
||||
|
|
|
|||
|
|
@ -83,7 +83,7 @@ struct ref *fetch_pack(struct fetch_pack_args *args,
|
|||
struct ref **sought,
|
||||
int nr_sought,
|
||||
struct oid_array *shallow,
|
||||
char **pack_lockfile,
|
||||
struct string_list *pack_lockfiles,
|
||||
enum protocol_version version);
|
||||
|
||||
/*
|
||||
|
|
|
|||
146
http-fetch.c
146
http-fetch.c
|
|
@ -5,59 +5,24 @@
|
|||
#include "walker.h"
|
||||
|
||||
static const char http_fetch_usage[] = "git http-fetch "
|
||||
"[-c] [-t] [-a] [-v] [--recover] [-w ref] [--stdin] commit-id url";
|
||||
"[-c] [-t] [-a] [-v] [--recover] [-w ref] [--stdin | --packfile=hash | commit-id] url";
|
||||
|
||||
int cmd_main(int argc, const char **argv)
|
||||
static int fetch_using_walker(const char *raw_url, int get_verbosely,
|
||||
int get_recover, int commits, char **commit_id,
|
||||
const char **write_ref, int commits_on_stdin)
|
||||
{
|
||||
struct walker *walker;
|
||||
int commits_on_stdin = 0;
|
||||
int commits;
|
||||
const char **write_ref = NULL;
|
||||
char **commit_id;
|
||||
char *url = NULL;
|
||||
int arg = 1;
|
||||
int rc = 0;
|
||||
int get_verbosely = 0;
|
||||
int get_recover = 0;
|
||||
struct walker *walker;
|
||||
int rc;
|
||||
|
||||
while (arg < argc && argv[arg][0] == '-') {
|
||||
if (argv[arg][1] == 't') {
|
||||
} else if (argv[arg][1] == 'c') {
|
||||
} else if (argv[arg][1] == 'a') {
|
||||
} else if (argv[arg][1] == 'v') {
|
||||
get_verbosely = 1;
|
||||
} else if (argv[arg][1] == 'w') {
|
||||
write_ref = &argv[arg + 1];
|
||||
arg++;
|
||||
} else if (argv[arg][1] == 'h') {
|
||||
usage(http_fetch_usage);
|
||||
} else if (!strcmp(argv[arg], "--recover")) {
|
||||
get_recover = 1;
|
||||
} else if (!strcmp(argv[arg], "--stdin")) {
|
||||
commits_on_stdin = 1;
|
||||
}
|
||||
arg++;
|
||||
}
|
||||
if (argc != arg + 2 - commits_on_stdin)
|
||||
usage(http_fetch_usage);
|
||||
if (commits_on_stdin) {
|
||||
commits = walker_targets_stdin(&commit_id, &write_ref);
|
||||
} else {
|
||||
commit_id = (char **) &argv[arg++];
|
||||
commits = 1;
|
||||
}
|
||||
|
||||
if (argv[arg])
|
||||
str_end_url_with_slash(argv[arg], &url);
|
||||
|
||||
setup_git_directory();
|
||||
|
||||
git_config(git_default_config, NULL);
|
||||
str_end_url_with_slash(raw_url, &url);
|
||||
|
||||
http_init(NULL, url, 0);
|
||||
|
||||
walker = get_http_walker(url);
|
||||
walker->get_verbosely = get_verbosely;
|
||||
walker->get_recover = get_recover;
|
||||
walker->get_progress = 0;
|
||||
|
||||
rc = walker_fetch(walker, commits, commit_id, write_ref, url);
|
||||
|
||||
|
|
@ -73,8 +38,99 @@ int cmd_main(int argc, const char **argv)
|
|||
|
||||
walker_free(walker);
|
||||
http_cleanup();
|
||||
|
||||
free(url);
|
||||
|
||||
return rc;
|
||||
}
|
||||
|
||||
static void fetch_single_packfile(struct object_id *packfile_hash,
|
||||
const char *url) {
|
||||
struct http_pack_request *preq;
|
||||
struct slot_results results;
|
||||
int ret;
|
||||
|
||||
http_init(NULL, url, 0);
|
||||
|
||||
preq = new_direct_http_pack_request(packfile_hash->hash, xstrdup(url));
|
||||
if (preq == NULL)
|
||||
die("couldn't create http pack request");
|
||||
preq->slot->results = &results;
|
||||
preq->generate_keep = 1;
|
||||
|
||||
if (start_active_slot(preq->slot)) {
|
||||
run_active_slot(preq->slot);
|
||||
if (results.curl_result != CURLE_OK) {
|
||||
die("Unable to get pack file %s\n%s", preq->url,
|
||||
curl_errorstr);
|
||||
}
|
||||
} else {
|
||||
die("Unable to start request");
|
||||
}
|
||||
|
||||
if ((ret = finish_http_pack_request(preq)))
|
||||
die("finish_http_pack_request gave result %d", ret);
|
||||
|
||||
release_http_pack_request(preq);
|
||||
http_cleanup();
|
||||
}
|
||||
|
||||
int cmd_main(int argc, const char **argv)
|
||||
{
|
||||
int commits_on_stdin = 0;
|
||||
int commits;
|
||||
const char **write_ref = NULL;
|
||||
char **commit_id;
|
||||
int arg = 1;
|
||||
int get_verbosely = 0;
|
||||
int get_recover = 0;
|
||||
int packfile = 0;
|
||||
struct object_id packfile_hash;
|
||||
|
||||
while (arg < argc && argv[arg][0] == '-') {
|
||||
const char *p;
|
||||
|
||||
if (argv[arg][1] == 't') {
|
||||
} else if (argv[arg][1] == 'c') {
|
||||
} else if (argv[arg][1] == 'a') {
|
||||
} else if (argv[arg][1] == 'v') {
|
||||
get_verbosely = 1;
|
||||
} else if (argv[arg][1] == 'w') {
|
||||
write_ref = &argv[arg + 1];
|
||||
arg++;
|
||||
} else if (argv[arg][1] == 'h') {
|
||||
usage(http_fetch_usage);
|
||||
} else if (!strcmp(argv[arg], "--recover")) {
|
||||
get_recover = 1;
|
||||
} else if (!strcmp(argv[arg], "--stdin")) {
|
||||
commits_on_stdin = 1;
|
||||
} else if (skip_prefix(argv[arg], "--packfile=", &p)) {
|
||||
const char *end;
|
||||
|
||||
packfile = 1;
|
||||
if (parse_oid_hex(p, &packfile_hash, &end) || *end)
|
||||
die(_("argument to --packfile must be a valid hash (got '%s')"), p);
|
||||
}
|
||||
arg++;
|
||||
}
|
||||
if (argc != arg + 2 - (commits_on_stdin || packfile))
|
||||
usage(http_fetch_usage);
|
||||
|
||||
setup_git_directory();
|
||||
|
||||
git_config(git_default_config, NULL);
|
||||
|
||||
if (packfile) {
|
||||
fetch_single_packfile(&packfile_hash, argv[arg]);
|
||||
return 0;
|
||||
}
|
||||
|
||||
if (commits_on_stdin) {
|
||||
commits = walker_targets_stdin(&commit_id, &write_ref);
|
||||
} else {
|
||||
commit_id = (char **) &argv[arg++];
|
||||
commits = 1;
|
||||
}
|
||||
return fetch_using_walker(argv[arg], get_verbosely, get_recover,
|
||||
commits, commit_id, write_ref,
|
||||
commits_on_stdin);
|
||||
}
|
||||
|
|
|
|||
|
|
@ -117,6 +117,7 @@ enum transfer_state {
|
|||
|
||||
struct transfer_request {
|
||||
struct object *obj;
|
||||
struct packed_git *target;
|
||||
char *url;
|
||||
char *dest;
|
||||
struct remote_lock *lock;
|
||||
|
|
@ -314,17 +315,18 @@ static void start_fetch_packed(struct transfer_request *request)
|
|||
release_request(request);
|
||||
return;
|
||||
}
|
||||
close_pack_index(target);
|
||||
request->target = target;
|
||||
|
||||
fprintf(stderr, "Fetching pack %s\n",
|
||||
hash_to_hex(target->hash));
|
||||
fprintf(stderr, " which contains %s\n", oid_to_hex(&request->obj->oid));
|
||||
|
||||
preq = new_http_pack_request(target, repo->url);
|
||||
preq = new_http_pack_request(target->hash, repo->url);
|
||||
if (preq == NULL) {
|
||||
repo->can_update_info_refs = 0;
|
||||
return;
|
||||
}
|
||||
preq->lst = &repo->packs;
|
||||
|
||||
/* Make sure there isn't another open request for this pack */
|
||||
while (check_request) {
|
||||
|
|
@ -597,6 +599,8 @@ static void finish_request(struct transfer_request *request)
|
|||
}
|
||||
if (fail)
|
||||
repo->can_update_info_refs = 0;
|
||||
else
|
||||
http_install_packfile(request->target, &repo->packs);
|
||||
release_request(request);
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -439,6 +439,7 @@ static int http_fetch_pack(struct walker *walker, struct alt_base *repo, unsigne
|
|||
target = find_sha1_pack(sha1, repo->packs);
|
||||
if (!target)
|
||||
return -1;
|
||||
close_pack_index(target);
|
||||
|
||||
if (walker->get_verbosely) {
|
||||
fprintf(stderr, "Getting pack %s\n",
|
||||
|
|
@ -447,10 +448,9 @@ static int http_fetch_pack(struct walker *walker, struct alt_base *repo, unsigne
|
|||
hash_to_hex(sha1));
|
||||
}
|
||||
|
||||
preq = new_http_pack_request(target, repo->base);
|
||||
preq = new_http_pack_request(target->hash, repo->base);
|
||||
if (preq == NULL)
|
||||
goto abort;
|
||||
preq->lst = &repo->packs;
|
||||
preq->slot->results = &results;
|
||||
|
||||
if (start_active_slot(preq->slot)) {
|
||||
|
|
@ -469,6 +469,7 @@ static int http_fetch_pack(struct walker *walker, struct alt_base *repo, unsigne
|
|||
release_http_pack_request(preq);
|
||||
if (ret)
|
||||
return ret;
|
||||
http_install_packfile(target, &repo->packs);
|
||||
|
||||
return 0;
|
||||
|
||||
|
|
|
|||
92
http.c
92
http.c
|
|
@ -2261,70 +2261,74 @@ void release_http_pack_request(struct http_pack_request *preq)
|
|||
|
||||
int finish_http_pack_request(struct http_pack_request *preq)
|
||||
{
|
||||
struct packed_git **lst;
|
||||
struct packed_git *p = preq->target;
|
||||
char *tmp_idx;
|
||||
size_t len;
|
||||
struct child_process ip = CHILD_PROCESS_INIT;
|
||||
|
||||
close_pack_index(p);
|
||||
int tmpfile_fd;
|
||||
int ret = 0;
|
||||
|
||||
fclose(preq->packfile);
|
||||
preq->packfile = NULL;
|
||||
|
||||
lst = preq->lst;
|
||||
tmpfile_fd = xopen(preq->tmpfile.buf, O_RDONLY);
|
||||
|
||||
argv_array_push(&ip.args, "index-pack");
|
||||
argv_array_push(&ip.args, "--stdin");
|
||||
ip.git_cmd = 1;
|
||||
ip.in = tmpfile_fd;
|
||||
if (preq->generate_keep) {
|
||||
argv_array_pushf(&ip.args, "--keep=git %"PRIuMAX,
|
||||
(uintmax_t)getpid());
|
||||
ip.out = 0;
|
||||
} else {
|
||||
ip.no_stdout = 1;
|
||||
}
|
||||
|
||||
if (run_command(&ip)) {
|
||||
ret = -1;
|
||||
goto cleanup;
|
||||
}
|
||||
|
||||
cleanup:
|
||||
close(tmpfile_fd);
|
||||
unlink(preq->tmpfile.buf);
|
||||
return ret;
|
||||
}
|
||||
|
||||
void http_install_packfile(struct packed_git *p,
|
||||
struct packed_git **list_to_remove_from)
|
||||
{
|
||||
struct packed_git **lst = list_to_remove_from;
|
||||
|
||||
while (*lst != p)
|
||||
lst = &((*lst)->next);
|
||||
*lst = (*lst)->next;
|
||||
|
||||
if (!strip_suffix(preq->tmpfile.buf, ".pack.temp", &len))
|
||||
BUG("pack tmpfile does not end in .pack.temp?");
|
||||
tmp_idx = xstrfmt("%.*s.idx.temp", (int)len, preq->tmpfile.buf);
|
||||
|
||||
argv_array_push(&ip.args, "index-pack");
|
||||
argv_array_pushl(&ip.args, "-o", tmp_idx, NULL);
|
||||
argv_array_push(&ip.args, preq->tmpfile.buf);
|
||||
ip.git_cmd = 1;
|
||||
ip.no_stdin = 1;
|
||||
ip.no_stdout = 1;
|
||||
|
||||
if (run_command(&ip)) {
|
||||
unlink(preq->tmpfile.buf);
|
||||
unlink(tmp_idx);
|
||||
free(tmp_idx);
|
||||
return -1;
|
||||
}
|
||||
|
||||
unlink(sha1_pack_index_name(p->hash));
|
||||
|
||||
if (finalize_object_file(preq->tmpfile.buf, sha1_pack_name(p->hash))
|
||||
|| finalize_object_file(tmp_idx, sha1_pack_index_name(p->hash))) {
|
||||
free(tmp_idx);
|
||||
return -1;
|
||||
}
|
||||
|
||||
install_packed_git(the_repository, p);
|
||||
free(tmp_idx);
|
||||
return 0;
|
||||
}
|
||||
|
||||
struct http_pack_request *new_http_pack_request(
|
||||
struct packed_git *target, const char *base_url)
|
||||
const unsigned char *packed_git_hash, const char *base_url) {
|
||||
|
||||
struct strbuf buf = STRBUF_INIT;
|
||||
|
||||
end_url_with_slash(&buf, base_url);
|
||||
strbuf_addf(&buf, "objects/pack/pack-%s.pack",
|
||||
hash_to_hex(packed_git_hash));
|
||||
return new_direct_http_pack_request(packed_git_hash,
|
||||
strbuf_detach(&buf, NULL));
|
||||
}
|
||||
|
||||
struct http_pack_request *new_direct_http_pack_request(
|
||||
const unsigned char *packed_git_hash, char *url)
|
||||
{
|
||||
off_t prev_posn = 0;
|
||||
struct strbuf buf = STRBUF_INIT;
|
||||
struct http_pack_request *preq;
|
||||
|
||||
preq = xcalloc(1, sizeof(*preq));
|
||||
strbuf_init(&preq->tmpfile, 0);
|
||||
preq->target = target;
|
||||
|
||||
end_url_with_slash(&buf, base_url);
|
||||
strbuf_addf(&buf, "objects/pack/pack-%s.pack",
|
||||
hash_to_hex(target->hash));
|
||||
preq->url = strbuf_detach(&buf, NULL);
|
||||
preq->url = url;
|
||||
|
||||
strbuf_addf(&preq->tmpfile, "%s.temp", sha1_pack_name(target->hash));
|
||||
strbuf_addf(&preq->tmpfile, "%s.temp", sha1_pack_name(packed_git_hash));
|
||||
preq->packfile = fopen(preq->tmpfile.buf, "a");
|
||||
if (!preq->packfile) {
|
||||
error("Unable to open local file %s for pack",
|
||||
|
|
@ -2348,7 +2352,7 @@ struct http_pack_request *new_http_pack_request(
|
|||
if (http_is_verbose)
|
||||
fprintf(stderr,
|
||||
"Resuming fetch of pack %s at byte %"PRIuMAX"\n",
|
||||
hash_to_hex(target->hash),
|
||||
hash_to_hex(packed_git_hash),
|
||||
(uintmax_t)prev_posn);
|
||||
http_opt_request_remainder(preq->slot->curl, prev_posn);
|
||||
}
|
||||
|
|
|
|||
24
http.h
24
http.h
|
|
@ -216,18 +216,36 @@ int http_get_info_packs(const char *base_url,
|
|||
|
||||
struct http_pack_request {
|
||||
char *url;
|
||||
struct packed_git *target;
|
||||
struct packed_git **lst;
|
||||
|
||||
/*
|
||||
* If this is true, finish_http_pack_request() will pass "--keep" to
|
||||
* index-pack, resulting in the creation of a keep file, and will not
|
||||
* suppress its stdout (that is, the "keep\t<hash>\n" line will be
|
||||
* printed to stdout).
|
||||
*/
|
||||
unsigned generate_keep : 1;
|
||||
|
||||
FILE *packfile;
|
||||
struct strbuf tmpfile;
|
||||
struct active_request_slot *slot;
|
||||
};
|
||||
|
||||
struct http_pack_request *new_http_pack_request(
|
||||
struct packed_git *target, const char *base_url);
|
||||
const unsigned char *packed_git_hash, const char *base_url);
|
||||
struct http_pack_request *new_direct_http_pack_request(
|
||||
const unsigned char *packed_git_hash, char *url);
|
||||
int finish_http_pack_request(struct http_pack_request *preq);
|
||||
void release_http_pack_request(struct http_pack_request *preq);
|
||||
|
||||
/*
|
||||
* Remove p from the given list, and invoke install_packed_git() on it.
|
||||
*
|
||||
* This is a convenience function for users that have obtained a list of packs
|
||||
* from http_get_info_packs() and have chosen a specific pack to fetch.
|
||||
*/
|
||||
void http_install_packfile(struct packed_git *p,
|
||||
struct packed_git **list_to_remove_from);
|
||||
|
||||
/* Helpers for fetching object */
|
||||
struct http_object_request {
|
||||
char *url;
|
||||
|
|
|
|||
|
|
@ -199,6 +199,28 @@ test_expect_success 'fetch packed objects' '
|
|||
git clone $HTTPD_URL/dumb/repo_pack.git
|
||||
'
|
||||
|
||||
test_expect_success 'http-fetch --packfile' '
|
||||
# Arbitrary hash. Use rev-parse so that we get one of the correct
|
||||
# length.
|
||||
ARBITRARY=$(git -C "$HTTPD_DOCUMENT_ROOT_PATH"/repo_pack.git rev-parse HEAD) &&
|
||||
|
||||
git init packfileclient &&
|
||||
p=$(cd "$HTTPD_DOCUMENT_ROOT_PATH"/repo_pack.git && ls objects/pack/pack-*.pack) &&
|
||||
git -C packfileclient http-fetch --packfile=$ARBITRARY "$HTTPD_URL"/dumb/repo_pack.git/$p >out &&
|
||||
|
||||
grep "^keep.[0-9a-f]\{16,\}$" out &&
|
||||
cut -c6- out >packhash &&
|
||||
|
||||
# Ensure that the expected files are generated
|
||||
test -e "packfileclient/.git/objects/pack/pack-$(cat packhash).pack" &&
|
||||
test -e "packfileclient/.git/objects/pack/pack-$(cat packhash).idx" &&
|
||||
test -e "packfileclient/.git/objects/pack/pack-$(cat packhash).keep" &&
|
||||
|
||||
# Ensure that it has the HEAD of repo_pack, at least
|
||||
HASH=$(git -C "$HTTPD_DOCUMENT_ROOT_PATH"/repo_pack.git rev-parse HEAD) &&
|
||||
git -C packfileclient cat-file -e "$HASH"
|
||||
'
|
||||
|
||||
test_expect_success 'fetch notices corrupt pack' '
|
||||
cp -R "$HTTPD_DOCUMENT_ROOT_PATH"/repo_pack.git "$HTTPD_DOCUMENT_ROOT_PATH"/repo_bad1.git &&
|
||||
(cd "$HTTPD_DOCUMENT_ROOT_PATH"/repo_bad1.git &&
|
||||
|
|
@ -214,6 +236,14 @@ test_expect_success 'fetch notices corrupt pack' '
|
|||
)
|
||||
'
|
||||
|
||||
test_expect_success 'http-fetch --packfile with corrupt pack' '
|
||||
rm -rf packfileclient &&
|
||||
git init packfileclient &&
|
||||
p=$(cd "$HTTPD_DOCUMENT_ROOT_PATH"/repo_bad1.git && ls objects/pack/pack-*.pack) &&
|
||||
test_must_fail git -C packfileclient http-fetch --packfile \
|
||||
"$HTTPD_URL"/dumb/repo_bad1.git/$p
|
||||
'
|
||||
|
||||
test_expect_success 'fetch notices corrupt idx' '
|
||||
cp -R "$HTTPD_DOCUMENT_ROOT_PATH"/repo_pack.git "$HTTPD_DOCUMENT_ROOT_PATH"/repo_bad2.git &&
|
||||
(cd "$HTTPD_DOCUMENT_ROOT_PATH"/repo_bad2.git &&
|
||||
|
|
|
|||
|
|
@ -795,6 +795,94 @@ test_expect_success 'when server does not send "ready", expect FLUSH' '
|
|||
test_i18ngrep "expected no other sections to be sent after no .ready." err
|
||||
'
|
||||
|
||||
configure_exclusion () {
|
||||
git -C "$1" hash-object "$2" >objh &&
|
||||
git -C "$1" pack-objects "$HTTPD_DOCUMENT_ROOT_PATH/mypack" <objh >packh &&
|
||||
git -C "$1" config --add \
|
||||
"uploadpack.blobpackfileuri" \
|
||||
"$(cat objh) $(cat packh) $HTTPD_URL/dumb/mypack-$(cat packh).pack" &&
|
||||
cat objh
|
||||
}
|
||||
|
||||
test_expect_success 'part of packfile response provided as URI' '
|
||||
P="$HTTPD_DOCUMENT_ROOT_PATH/http_parent" &&
|
||||
rm -rf "$P" http_child log &&
|
||||
|
||||
git init "$P" &&
|
||||
git -C "$P" config "uploadpack.allowsidebandall" "true" &&
|
||||
|
||||
echo my-blob >"$P/my-blob" &&
|
||||
git -C "$P" add my-blob &&
|
||||
echo other-blob >"$P/other-blob" &&
|
||||
git -C "$P" add other-blob &&
|
||||
git -C "$P" commit -m x &&
|
||||
|
||||
configure_exclusion "$P" my-blob >h &&
|
||||
configure_exclusion "$P" other-blob >h2 &&
|
||||
|
||||
GIT_TRACE=1 GIT_TRACE_PACKET="$(pwd)/log" GIT_TEST_SIDEBAND_ALL=1 \
|
||||
git -c protocol.version=2 \
|
||||
-c fetch.uriprotocols=http,https \
|
||||
clone "$HTTPD_URL/smart/http_parent" http_child &&
|
||||
|
||||
# Ensure that my-blob and other-blob are in separate packfiles.
|
||||
for idx in http_child/.git/objects/pack/*.idx
|
||||
do
|
||||
git verify-pack --verbose $idx >out &&
|
||||
{
|
||||
grep "^[0-9a-f]\{16,\} " out || :
|
||||
} >out.objectlist &&
|
||||
if test_line_count = 1 out.objectlist
|
||||
then
|
||||
if grep $(cat h) out
|
||||
then
|
||||
>hfound
|
||||
fi &&
|
||||
if grep $(cat h2) out
|
||||
then
|
||||
>h2found
|
||||
fi
|
||||
fi
|
||||
done &&
|
||||
test -f hfound &&
|
||||
test -f h2found &&
|
||||
|
||||
# Ensure that there are exactly 6 files (3 .pack and 3 .idx).
|
||||
ls http_child/.git/objects/pack/* >filelist &&
|
||||
test_line_count = 6 filelist
|
||||
'
|
||||
|
||||
test_expect_success 'fetching with valid packfile URI but invalid hash fails' '
|
||||
P="$HTTPD_DOCUMENT_ROOT_PATH/http_parent" &&
|
||||
rm -rf "$P" http_child log &&
|
||||
|
||||
git init "$P" &&
|
||||
git -C "$P" config "uploadpack.allowsidebandall" "true" &&
|
||||
|
||||
echo my-blob >"$P/my-blob" &&
|
||||
git -C "$P" add my-blob &&
|
||||
echo other-blob >"$P/other-blob" &&
|
||||
git -C "$P" add other-blob &&
|
||||
git -C "$P" commit -m x &&
|
||||
|
||||
configure_exclusion "$P" my-blob >h &&
|
||||
# Configure a URL for other-blob. Just reuse the hash of the object as
|
||||
# the hash of the packfile, since the hash does not matter for this
|
||||
# test as long as it is not the hash of the pack, and it is of the
|
||||
# expected length.
|
||||
git -C "$P" hash-object other-blob >objh &&
|
||||
git -C "$P" pack-objects "$HTTPD_DOCUMENT_ROOT_PATH/mypack" <objh >packh &&
|
||||
git -C "$P" config --add \
|
||||
"uploadpack.blobpackfileuri" \
|
||||
"$(cat objh) $(cat objh) $HTTPD_URL/dumb/mypack-$(cat packh).pack" &&
|
||||
|
||||
test_must_fail env GIT_TEST_SIDEBAND_ALL=1 \
|
||||
git -c protocol.version=2 \
|
||||
-c fetch.uriprotocols=http,https \
|
||||
clone "$HTTPD_URL/smart/http_parent" http_child 2>err &&
|
||||
test_i18ngrep "pack downloaded from.*does not match expected hash" err
|
||||
'
|
||||
|
||||
# DO NOT add non-httpd-specific tests here, because the last part of this
|
||||
# test script is only executed when httpd is available and enabled.
|
||||
|
||||
|
|
|
|||
|
|
@ -410,10 +410,11 @@ static int fetch_with_fetch(struct transport *transport,
|
|||
exit(128);
|
||||
|
||||
if (skip_prefix(buf.buf, "lock ", &name)) {
|
||||
if (transport->pack_lockfile)
|
||||
if (transport->pack_lockfiles.nr)
|
||||
warning(_("%s also locked %s"), data->name, name);
|
||||
else
|
||||
transport->pack_lockfile = xstrdup(name);
|
||||
string_list_append(&transport->pack_lockfiles,
|
||||
name);
|
||||
}
|
||||
else if (data->check_connectivity &&
|
||||
data->transport_options.check_self_contained_and_connected &&
|
||||
|
|
|
|||
12
transport.c
12
transport.c
|
|
@ -378,7 +378,7 @@ static int fetch_refs_via_pack(struct transport *transport,
|
|||
refs = fetch_pack(&args, data->fd,
|
||||
refs_tmp ? refs_tmp : transport->remote_refs,
|
||||
to_fetch, nr_heads, &data->shallow,
|
||||
&transport->pack_lockfile, data->version);
|
||||
&transport->pack_lockfiles, data->version);
|
||||
|
||||
close(data->fd[0]);
|
||||
close(data->fd[1]);
|
||||
|
|
@ -921,6 +921,7 @@ struct transport *transport_get(struct remote *remote, const char *url)
|
|||
struct transport *ret = xcalloc(1, sizeof(*ret));
|
||||
|
||||
ret->progress = isatty(2);
|
||||
string_list_init(&ret->pack_lockfiles, 1);
|
||||
|
||||
if (!remote)
|
||||
BUG("No remote provided to transport_get()");
|
||||
|
|
@ -1316,10 +1317,11 @@ int transport_fetch_refs(struct transport *transport, struct ref *refs)
|
|||
|
||||
void transport_unlock_pack(struct transport *transport)
|
||||
{
|
||||
if (transport->pack_lockfile) {
|
||||
unlink_or_warn(transport->pack_lockfile);
|
||||
FREE_AND_NULL(transport->pack_lockfile);
|
||||
}
|
||||
int i;
|
||||
|
||||
for (i = 0; i < transport->pack_lockfiles.nr; i++)
|
||||
unlink_or_warn(transport->pack_lockfiles.items[i].string);
|
||||
string_list_clear(&transport->pack_lockfiles, 0);
|
||||
}
|
||||
|
||||
int transport_connect(struct transport *transport, const char *name,
|
||||
|
|
|
|||
|
|
@ -5,8 +5,7 @@
|
|||
#include "run-command.h"
|
||||
#include "remote.h"
|
||||
#include "list-objects-filter-options.h"
|
||||
|
||||
struct string_list;
|
||||
#include "string-list.h"
|
||||
|
||||
struct git_transport_options {
|
||||
unsigned thin : 1;
|
||||
|
|
@ -98,7 +97,8 @@ struct transport {
|
|||
*/
|
||||
const struct string_list *server_options;
|
||||
|
||||
char *pack_lockfile;
|
||||
struct string_list pack_lockfiles;
|
||||
|
||||
signed verbose : 3;
|
||||
/**
|
||||
* Transports should not set this directly, and should use this
|
||||
|
|
|
|||
156
upload-pack.c
156
upload-pack.c
|
|
@ -84,6 +84,7 @@ struct upload_pack_data {
|
|||
/* 0 for no sideband, otherwise DEFAULT_PACKET_MAX or LARGE_PACKET_MAX */
|
||||
int use_sideband;
|
||||
|
||||
struct string_list uri_protocols;
|
||||
enum allow_uor allow_uor;
|
||||
|
||||
struct list_objects_filter_options filter_options;
|
||||
|
|
@ -117,6 +118,7 @@ static void upload_pack_data_init(struct upload_pack_data *data)
|
|||
struct oid_array haves = OID_ARRAY_INIT;
|
||||
struct object_array shallows = OBJECT_ARRAY_INIT;
|
||||
struct string_list deepen_not = STRING_LIST_INIT_DUP;
|
||||
struct string_list uri_protocols = STRING_LIST_INIT_DUP;
|
||||
struct object_array extra_edge_obj = OBJECT_ARRAY_INIT;
|
||||
|
||||
memset(data, 0, sizeof(*data));
|
||||
|
|
@ -127,6 +129,7 @@ static void upload_pack_data_init(struct upload_pack_data *data)
|
|||
data->haves = haves;
|
||||
data->shallows = shallows;
|
||||
data->deepen_not = deepen_not;
|
||||
data->uri_protocols = uri_protocols;
|
||||
data->extra_edge_obj = extra_edge_obj;
|
||||
packet_writer_init(&data->writer, 1);
|
||||
|
||||
|
|
@ -179,13 +182,86 @@ static int write_one_shallow(const struct commit_graft *graft, void *cb_data)
|
|||
return 0;
|
||||
}
|
||||
|
||||
static void create_pack_file(struct upload_pack_data *pack_data)
|
||||
struct output_state {
|
||||
char buffer[8193];
|
||||
int used;
|
||||
unsigned packfile_uris_started : 1;
|
||||
unsigned packfile_started : 1;
|
||||
};
|
||||
|
||||
static int relay_pack_data(int pack_objects_out, struct output_state *os,
|
||||
int use_sideband, int write_packfile_line)
|
||||
{
|
||||
/*
|
||||
* We keep the last byte to ourselves
|
||||
* in case we detect broken rev-list, so that we
|
||||
* can leave the stream corrupted. This is
|
||||
* unfortunate -- unpack-objects would happily
|
||||
* accept a valid packdata with trailing garbage,
|
||||
* so appending garbage after we pass all the
|
||||
* pack data is not good enough to signal
|
||||
* breakage to downstream.
|
||||
*/
|
||||
ssize_t readsz;
|
||||
|
||||
readsz = xread(pack_objects_out, os->buffer + os->used,
|
||||
sizeof(os->buffer) - os->used);
|
||||
if (readsz < 0) {
|
||||
return readsz;
|
||||
}
|
||||
os->used += readsz;
|
||||
|
||||
while (!os->packfile_started) {
|
||||
char *p;
|
||||
if (os->used >= 4 && !memcmp(os->buffer, "PACK", 4)) {
|
||||
os->packfile_started = 1;
|
||||
if (write_packfile_line) {
|
||||
if (os->packfile_uris_started)
|
||||
packet_delim(1);
|
||||
packet_write_fmt(1, "\1packfile\n");
|
||||
}
|
||||
break;
|
||||
}
|
||||
if ((p = memchr(os->buffer, '\n', os->used))) {
|
||||
if (!os->packfile_uris_started) {
|
||||
os->packfile_uris_started = 1;
|
||||
if (!write_packfile_line)
|
||||
BUG("packfile_uris requires sideband-all");
|
||||
packet_write_fmt(1, "\1packfile-uris\n");
|
||||
}
|
||||
*p = '\0';
|
||||
packet_write_fmt(1, "\1%s\n", os->buffer);
|
||||
|
||||
os->used -= p - os->buffer + 1;
|
||||
memmove(os->buffer, p + 1, os->used);
|
||||
} else {
|
||||
/*
|
||||
* Incomplete line.
|
||||
*/
|
||||
return readsz;
|
||||
}
|
||||
}
|
||||
|
||||
if (os->used > 1) {
|
||||
send_client_data(1, os->buffer, os->used - 1, use_sideband);
|
||||
os->buffer[0] = os->buffer[os->used - 1];
|
||||
os->used = 1;
|
||||
} else {
|
||||
send_client_data(1, os->buffer, os->used, use_sideband);
|
||||
os->used = 0;
|
||||
}
|
||||
|
||||
return readsz;
|
||||
}
|
||||
|
||||
static void create_pack_file(struct upload_pack_data *pack_data,
|
||||
const struct string_list *uri_protocols)
|
||||
{
|
||||
struct child_process pack_objects = CHILD_PROCESS_INIT;
|
||||
char data[8193], progress[128];
|
||||
struct output_state output_state = { { 0 } };
|
||||
char progress[128];
|
||||
char abort_msg[] = "aborting due to possible repository "
|
||||
"corruption on the remote side.";
|
||||
int buffered = -1;
|
||||
ssize_t sz;
|
||||
int i;
|
||||
FILE *pipe_fd;
|
||||
|
|
@ -229,6 +305,11 @@ static void create_pack_file(struct upload_pack_data *pack_data)
|
|||
spec);
|
||||
}
|
||||
}
|
||||
if (uri_protocols) {
|
||||
for (i = 0; i < uri_protocols->nr; i++)
|
||||
argv_array_pushf(&pack_objects.args, "--uri-protocol=%s",
|
||||
uri_protocols->items[i].string);
|
||||
}
|
||||
|
||||
pack_objects.in = -1;
|
||||
pack_objects.out = -1;
|
||||
|
|
@ -318,40 +399,17 @@ static void create_pack_file(struct upload_pack_data *pack_data)
|
|||
continue;
|
||||
}
|
||||
if (0 <= pu && (pfd[pu].revents & (POLLIN|POLLHUP))) {
|
||||
/* Data ready; we keep the last byte to ourselves
|
||||
* in case we detect broken rev-list, so that we
|
||||
* can leave the stream corrupted. This is
|
||||
* unfortunate -- unpack-objects would happily
|
||||
* accept a valid packdata with trailing garbage,
|
||||
* so appending garbage after we pass all the
|
||||
* pack data is not good enough to signal
|
||||
* breakage to downstream.
|
||||
*/
|
||||
char *cp = data;
|
||||
ssize_t outsz = 0;
|
||||
if (0 <= buffered) {
|
||||
*cp++ = buffered;
|
||||
outsz++;
|
||||
}
|
||||
sz = xread(pack_objects.out, cp,
|
||||
sizeof(data) - outsz);
|
||||
if (0 < sz)
|
||||
;
|
||||
else if (sz == 0) {
|
||||
int result = relay_pack_data(pack_objects.out,
|
||||
&output_state,
|
||||
pack_data->use_sideband,
|
||||
!!uri_protocols);
|
||||
|
||||
if (result == 0) {
|
||||
close(pack_objects.out);
|
||||
pack_objects.out = -1;
|
||||
}
|
||||
else
|
||||
} else if (result < 0) {
|
||||
goto fail;
|
||||
sz += outsz;
|
||||
if (1 < sz) {
|
||||
buffered = data[sz-1] & 0xFF;
|
||||
sz--;
|
||||
}
|
||||
else
|
||||
buffered = -1;
|
||||
send_client_data(1, data, sz,
|
||||
pack_data->use_sideband);
|
||||
}
|
||||
|
||||
/*
|
||||
|
|
@ -376,9 +434,8 @@ static void create_pack_file(struct upload_pack_data *pack_data)
|
|||
}
|
||||
|
||||
/* flush the data */
|
||||
if (0 <= buffered) {
|
||||
data[0] = buffered;
|
||||
send_client_data(1, data, 1,
|
||||
if (output_state.used > 0) {
|
||||
send_client_data(1, output_state.buffer, output_state.used,
|
||||
pack_data->use_sideband);
|
||||
fprintf(stderr, "flushed.\n");
|
||||
}
|
||||
|
|
@ -1188,7 +1245,7 @@ void upload_pack(struct upload_pack_options *options)
|
|||
receive_needs(&data, &reader);
|
||||
if (data.want_obj.nr) {
|
||||
get_common_commits(&data, &reader);
|
||||
create_pack_file(&data);
|
||||
create_pack_file(&data, NULL);
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -1341,10 +1398,18 @@ static void process_args(struct packet_reader *request,
|
|||
continue;
|
||||
}
|
||||
|
||||
if (skip_prefix(arg, "packfile-uris ", &p)) {
|
||||
string_list_split(&data->uri_protocols, p, ',', -1);
|
||||
continue;
|
||||
}
|
||||
|
||||
/* ignore unknown lines maybe? */
|
||||
die("unexpected line: '%s'", arg);
|
||||
}
|
||||
|
||||
if (data->uri_protocols.nr && !data->writer.use_sideband)
|
||||
string_list_clear(&data->uri_protocols, 0);
|
||||
|
||||
if (request->status != PACKET_READ_FLUSH)
|
||||
die(_("expected flush after fetch arguments"));
|
||||
}
|
||||
|
|
@ -1502,8 +1567,12 @@ int upload_pack_v2(struct repository *r, struct argv_array *keys,
|
|||
send_wanted_ref_info(&data);
|
||||
send_shallow_info(&data);
|
||||
|
||||
packet_writer_write(&data.writer, "packfile\n");
|
||||
create_pack_file(&data);
|
||||
if (data.uri_protocols.nr) {
|
||||
create_pack_file(&data, &data.uri_protocols);
|
||||
} else {
|
||||
packet_writer_write(&data.writer, "packfile\n");
|
||||
create_pack_file(&data, NULL);
|
||||
}
|
||||
state = FETCH_DONE;
|
||||
break;
|
||||
case FETCH_DONE:
|
||||
|
|
@ -1522,6 +1591,7 @@ int upload_pack_advertise(struct repository *r,
|
|||
int allow_filter_value;
|
||||
int allow_ref_in_want;
|
||||
int allow_sideband_all_value;
|
||||
char *str = NULL;
|
||||
|
||||
strbuf_addstr(value, "shallow");
|
||||
|
||||
|
|
@ -1543,6 +1613,14 @@ int upload_pack_advertise(struct repository *r,
|
|||
&allow_sideband_all_value) &&
|
||||
allow_sideband_all_value))
|
||||
strbuf_addstr(value, " sideband-all");
|
||||
|
||||
if (!repo_config_get_string(the_repository,
|
||||
"uploadpack.blobpackfileuri",
|
||||
&str) &&
|
||||
str) {
|
||||
strbuf_addstr(value, " packfile-uris");
|
||||
free(str);
|
||||
}
|
||||
}
|
||||
|
||||
return 1;
|
||||
|
|
|
|||
Loading…
Reference in New Issue