|
Unix VPS
Support
FAQ
Network
Miscellaneous
|
| GIT-FILTER-REPO(1) | Git Manual | GIT-FILTER-REPO(1) |
NAME
git-filter-repo - Rewrite repository historySYNOPSIS
git filter-repo --analyze
git filter-repo [<path_filtering_options>] [<content_filtering_options>]
[<ref_renaming_options>] [<commit_message_filtering_options>]
[<name_or_email_filtering_options>] [<parent_rewriting_options>]
[<generic_callback_options>] [<miscellaneous_options>]
DESCRIPTION
Rapidly rewrite entire repository history using user-specified filters. This is a destructive operation which should not be used lightly; it writes new commits, trees, tags, and blobs corresponding to (but filtered from) the original objects in the repository, then deletes the original history and leaves only the new. See the section called “DISCUSSION” for more details on the ramifications of using this tool. Several different types of history rewrites are possible; examples include (but are not limited to):Additionally, several concerns are handled automatically (many of these can be overridden, but they are all on by default):
Also, it’s worth noting that there is an important safety mechanism:
For those who know that there is large unwanted stuff in their history and want help finding it, this command also
See also the section called “VERSATILITY”, the section called “DISCUSSION”, the section called “EXAMPLES”, and the section called “INTERNALS”.
OPTIONS
Analysis Options
--analyzeFiltering based on paths (see also --filename-callback)
--invert-paths--path-match <dir_or_file>, --path <dir_or_file>
--path-glob <glob>
--path-regex <regex>
--use-base-name
Renaming based on paths (see also --filename-callback)
Note: if you combine path filtering with path renaming, be aware that a rename directive does not select paths, it only says how to rename paths that are selected with the filters.--path-rename <old_name:new_name>, --path-rename-match <old_name:new_name>
Path shortcuts
--paths-from-file <filename>--subdirectory-filter <directory>
--to-subdirectory-filter <directory>
Content editing filters (see also --blob-callback)
--replace-text <expressions_file>--strip-blobs-bigger-than <size>
--strip-blobs-with-ids <blob_id_filename>
Renaming of refs (see also --refname-callback)
--tag-rename <old:new>Filtering of commit messages (see also --message-callback)
--preserve-commit-hashes--preserve-commit-encoding
Filtering of names & emails (see also --name-callback and --email-callback)
--mailmap <filename>--use-mailmap
Parent rewriting
--replace-refs {delete-no-add, delete-and-add, update-no-add, update-or-add, update-and-add}--prune-empty {always, auto, never}
--prune-degenerate {always, auto, never}
--no-ff
Generic callback code snippets
--filename-callback <function_body>--message-callback <function_body>
--name-callback <function_body>
--email-callback <function_body>
--refname-callback <function_body>
--blob-callback <function_body>
--commit-callback <function_body>
--tag-callback <function_body>
--reset-callback <function_body>
Location to filter from/to
Note
Specifying alternate source or target locations implies --partial except that the normal default for --replace-refs is used. However, unlike normal uses of --partial, this doesn’t risk mixing old and new history since the old and new histories are in different repositories.
--source <source>
--target <target>
Miscellaneous options
--help, -h--force, -f
--partial
--refs <refs+>
--dry-run
--debug
--stdin
--quiet
OUTPUT
Every time filter-repo is run, files are created in the .git/filter-repo/ directory. These files overwritten unconditionally on every run.Commit map
The .git/filter-repo/commit-map file contains a mapping of how all commits were (or were not) changed.Reference map
The .git/filter-repo/ref-map file contains a mapping of which local references were changed.FRESH CLONE SAFETY CHECK AND --FORCE
Since filter-repo does irreversible rewriting of history, it is important to avoid making changes to a repo for which the user doesn’t have a good backup. The primary defense mechanism is to simply educate users and rely on them to be good stewards of their data; thus there are several warnings in the documentation about how filter repo rewrites history.However, as a service to users, we would like to provide an additional safety check beyond the documentation. There isn’t a good way to check if the user has a good backup, but we can ask a related question that is an imperfect but quite reasonable proxy: "Is this repository a fresh clone?" Unfortunately, that is also a question we can’t get a perfect answer to; git provides no way to answer that question. However, there are approximately a dozen things that I found that seem to always be true of brand new clones (assuming they are either clones of remote repositories or are made with the --no-local flag), and I check for all of those.
These checks can have both false positives and false negatives. Someone might have a perfectly good backup of their repo without it actually being a fresh clone — but there’s no way for filter-repo to know that. Conversely, someone could look at all things that filter-repo checks for in its safety checks and then just tweak their non-backed-up repository to satisfy those conditions (though it would take a fair amount of effort, and it’s astronomically unlikely that a repo that isn’t a fresh clone randomly happens to match all the criteria). In practice, the safety checks filter-repo uses seem to be really good at avoiding people accidentally running filter-repo on a repository that they shouldn’t be running it on. It even caught me once when I did mean to run filter-repo but was in a different directory than I thought I was.
In short, it’s perfectly fine to use ‘--force` to override the safety checks as long as you’re okay with filter-repo irreversibly rewriting the contents of the current repository. It is a really bad idea to get in the habit of always specifying --force; if you do, one day you will run one of your commands in the wrong directory like I did, and you won’t have the safety check anymore to bail you out. Also, it is definitely NOT okay to recommend --force on forums, Q&A sites, or in emails to other users without first carefully explaining that --force means putting your repositories’ data at risk. I am especially bothered by people who suggest the flag when it clearly is NOT needed; they are needlessly putting other peoples' data at risk.
VERSATILITY
filter-repo has a hierarchy of capabilities on the spectrum from easy to use convenience flags that perform pre-defined types of filtering, to choices that provide lots of flexibility in controlling how filtering occurs. This spectrum includes the following:For more information about callbacks, see the section called “CALLBACKS”. For examples on writing python programs that import filter-repo as a module to create new history rewriting tools, look at the contrib/filter-repo-demos/ directory. That directory includes, among other examples, a reimplementation of git-filter-branch which is faster than git-filter-branch, and a reimplementation of BFG Repo Cleaner with several bug fixes and new features.
DISCUSSION
Using filter-repo is relatively simple, but rewriting history is part of a larger discussion in terms of collaboration. When you rewrite history, the old and new histories are no longer compatible; if you push this history somewhere for others to view, it will look as though you’ve done a rebase of all branches and tags. Make sure you are familiar with the "RECOVERING FROM UPSTREAM REBASE" section of git-rebase(1) (and in particular, "The hard case") before proceeding, in addition to this section.Steps to use git-filter-repo as part of the bigger picture of doing a history rewrite are roughly as follows:
EXAMPLES
Path based filtering
To only keep the README.md file plus the directories guides and tools/releases/:git filter-repo --path README.md --path guides/ --path tools/releases
Directory names can be given with or without a trailing slash, and all filenames are relative to the toplevel of the repo. To keep all files except these paths, just add --invert-paths:
git filter-repo --path README.md --path guides/ --path tools/releases --invert-paths
If you want to have both an inclusion filter and an exclusion filter, just run filter-repo multiple times. For example, to keep the src/main subdirectory but exclude files under src/main named data, run:
git filter-repo --path src/main/ git filter-repo --path-glob 'src/*/data' --invert-paths
Note that the asterisk (*) will match across multiple directories, so the second command would remove e.g. src/main/org/whatever/data. Also, the second command by itself would also remove e.g. src/not-main/foo/data, but since src/not-main/ was removed by the first command, that’s not an issue. Also, the use of quotes around the asterisk is sometimes important to avoid glob expansion by the shell.
You can also select paths by regular expression (see https://docs.python.org/3/library/re.html#regular-expression-syntax). For example, to only include files from the repo whose name is in the format YYYY-MM-DD.txt and is found at least two subdirectories deep:
git filter-repo --path-regex '^.*/.*/[0-9]{4}-[0-9]{2}-[0-9]{2}.txt$'
If you want two directories to be renamed (and maybe merged if both are renamed to the same location), use --path-rename; for example, to rename both cmds/ and src/scripts/ to tools/:
git filter-repo --path-rename cmds:tools --path-rename src/scripts/:tools/
As with --path, directories can be specified with or without a trailing slash for --path-rename.
If you do a --path-rename to something that was already in use, it will be silently overwritten. However, if you try to rename multiple files to the same location (e.g. src/scripts/run_release.sh and cmds/run_release.sh both existed and had different content with the renames above), then you will be given an error. If you have such a case, you may want to add another rename command to move one of the paths somewhere else where it won’t collide:
git filter-repo --path-rename cmds/run_release.sh:tools/do_release.sh \
--path-rename cmds/:tools/ \
--path-rename src/scripts/:tools/
Also, --path-rename brings up ordering issues; all path arguments are applied in order. Thus, a command like
git filter-repo --path-rename sources/:src/main/ --path src/main/
would make sense but reversing the two arguments would not (src/main/ is created by the rename so reversing the two would give you an empty repo). Also, note that the rename of cmds/run_release.sh a couple examples ago was done before the other renames.
Note that path renaming does not do path filtering, thus the following command
git filter-repo --path src/main/ --path-rename tools/:scripts/
would not result in the tools or scripts directories being present, because the single filter selected only src/main/. It’s likely that you would instead want to run:
git filter-repo --path src/main/ --path tools/ --path-rename tools/:scripts/
If you prefer to filter based solely on basename, use the --use-base-name flag (though this is incompatible with --path-rename). For example, to only include README.md and Makefile files from any directory:
git filter-repo --use-base-name --path README.md --path Makefile
If you wanted to delete all .DS_Store files in any directory, you could either use:
git filter-repo --invert-paths --path '.DS_Store' --use-base-name
or
git filter-repo --invert-paths --path-glob '*/.DS_Store' --path '.DS_Store'
(the --path-glob isn’t sufficient by itself as it might miss a toplevel .DS_Store file; further while something like --path-glob '*.DS_Store' would workaround that problem it would also grab files named foo.DS_Store or bar/baz.DS_Store)
Finally, see also the --filename-callback from the section called “CALLBACKS”.
Filtering based on many paths
If you have a long list of files, directories, globs, or regular expressions to filter on, you can stick them in a file and use --paths-from-file; for example, with a file named stuff-i-want.txt with contents of
# Blank lines and comment lines are ignored.
# Examples similar to --path:
README.md
guides/
tools/releases
# An example that is like --path-glob:
glob:*.py
# An example that is like --path-regex:
regex:^.*/.*/[0-9]{4}-[0-9]{2}-[0-9]{2}.txt$
# An example of renaming a path
tools/==>scripts/
# An example of using a regex to rename a path
regex:(.*)/([^/]*)/([^/]*)\.text$==>\2/\1/\3.txt
then you could run
git filter-repo --paths-from-file stuff-i-want.txt
to get a repo containing only the toplevel README.md file, the guides/ and tools/releases/ directories, all python files, files whose name was of the form YYYY.MM-DD.txt at least two subdirectories deep, and would rename tools/ to scripts/ and rename files like foo/bar/baz.text to bar/foo/baz.txt. Note the special line prefixes of glob: and regex: and the special string ==> denoting renames.
Sometimes you have a way of easily generating all the files you want. For example, if you know that none of the currently tracked files have any newlines or special characters in them (see core.quotePath from git config --help) so that git ls-files would print all files literally one per line, and you knew that you wanted to keep only the files that are currently tracked (thus deleting from all commits in history any files that only appear on other branches or that only appear in older commits), then you could use a pair of commands such as
git ls-files >../paths-i-want.txt git filter-repo --paths-from-file ../paths-i-want.txt
Similarly, you could use --paths-from-file to delete many files. For example, you could run git filter-repo --analyze to get reports, look in one such as .git/filter-repo/analysis/path-deleted-sizes.txt and copy all the filenames into a file such as /tmp/files-i-dont-want-anymore.txt and then run
git filter-repo --invert-paths --paths-from-file /tmp/files-i-dont-want-anymore.txt
to delete them all.
Directory based shortcuts
Let’s say you had a directory structure like the following:module/ foo.c bar.c otherDir/ blah.config stuff.txt zebra.jpg
If you wanted just the module/ directory and you wanted it to become the new root so that your new directory structure looked like
foo.c bar.c
then you could run:
git filter-repo --subdirectory-filter module/
If you wanted all the files from the original repo, but wanted to move everything under a subdirectory named my-module/, so that your new directory structure looked like
my-module/
module/
foo.c
bar.c
otherDir/
blah.config
stuff.txt
zebra.jpg
then you would instead run run
git filter-repo --to-subdirectory-filter my-module/
Content based filtering
If you want to filter out all files bigger than a certain size, you can use --strip-blobs-bigger-than with some size (K, M, and G suffixes are recognized), e.g.:git filter-repo --strip-blobs-bigger-than 10M
If you want to strip out all files with specified git object ids (hashes), list the hashes in a file and run
git filter-repo --strip-blobs-with-ids FILE_WITH_GIT_BLOB_IDS
If you want to modify file contents, you can do so based on a list of expressions in a file, one per line. For example, with a file named expressions.txt containing
p455w0rd
foo==>bar
glob:*666*==>
regex:\bdriver\b==>pilot
literal:MM/DD/YYYY==>YYYY-MM-DD
regex:([0-9]{2})/([0-9]{2})/([0-9]{4})==>\3-\1-\2
then running
git filter-repo --replace-text expressions.txt
will go through and replace p455w0rd with ***REMOVED***, foo with bar, any line containing 666 with a blank line, the word driver with pilot (but not if it has letters before or after; e.g. drivers will be unmodified), replace the exact text MM/DD/YYYY with YYYY-MM-DD and replace date strings of the form MM/DD/YYYY with ones of the form YYYY-MM-DD. In the expressions file, there are a few things to note:
See also the --blob-callback from the section called “CALLBACKS”.
Refname based filtering
To rename tags, use --tag-rename, e.g.:git filter-repo --tag-rename foo:bar
This will rename any tags starting with foo to now start with bar. Either side of the colon could be blank, e.g.
git filter-repo --tag-rename '':'my-module-'
For more general refname modification, see --refname-callback from the section called “CALLBACKS”.
User and email based filtering
To modify username and emails of commits, you can create a mailmap file in the format accepted by git-shortlog(1). For example, if you have a file named my-mailmap you can rungit filter-repo --mailmap my-mailmap
and if the current contents of that file are as follows (if the specified mailmap file is version controlled, historical versions of the file are ignored):
Name For User <email@addre.ss> <new@ema.il> <old1@ema.il> New Name And <new@ema.il> <old2@ema.il> New Name And <new@ema.il> Old Name And <old3@ema.il>
then we can update username and/or emails based on the specified mapping.
See also the --name-callback and --email-callback from the section called “CALLBACKS”.
Parent rewriting
To replace $commit_A with $commit_B (e.g. make all commits which had $commit_A as a parent instead have $commit_B for that parent), and rewrite history to make it permanent:git replace $commit_A $commit_B git filter-repo --force
To create a new commit with the same contents as $commit_A except with different parent(s) and then replace $commit_A with the new commit, and rewrite history to make it permanent:
git replace --graft $commit_A $new_parent_or_parents git filter-repo --force
The reason to specify --force is two-fold: filter-repo will error out if no arguments are specified, and the new graft commit would otherwise trigger the not-a-fresh-clone check.
Partial history rewrites
To rewrite the history on just one branch (which may cause it to no longer share any common history with other branches), use --refs. For example, to remove a file named extraneous.txt from the master branch:git filter-repo --invert-paths --path extraneous.txt --refs master
To rewrite just some recent commits:
git filter-repo --invert-paths --path extraneous.txt --refs master~3..master
CALLBACKS
For flexibility, filter-repo allows you to specify functions on the command line to further filter all changes. Please note that there are some API compatibility caveats associated with these callbacks that you should be aware of before using them; see the "API BACKWARD COMPATIBILITY CAVEAT" comment near the top of git-filter-repo source code.All callback functions are of the same general format. For a command line argument like
--foo-callback 'BODY'
the following code will be compiled and called:
def foo_callback(foo): BODY
Thus, you just need to make sure your BODY modifies and returns foo appropriately. One important thing to note for all callbacks is that filter-repo uses bytestrings (see https://docs.python.org/3/library/stdtypes.html#bytes) everywhere instead of strings.
There are four callbacks that allow you to operate directly on raw objects that contain data that’s easy to write in fast-import(1) format:
--blob-callback --commit-callback --tag-callback --reset-callback
We’ll come back to these later because it is often the case that the other callbacks are more convenient. The other callbacks operate on a small piece of the raw objects or operate on pieces across multiple types of raw object (e.g. author names and committer names and tagger names across commits and tags, or refnames across commits, tags, and resets, or messages across commits and tags). The convenience callbacks are:
--filename-callback --message-callback --name-callback --email-callback --refname-callback
in each you are expected to simply return a new value based on the one passed in. For example,
git-filter-repo --name-callback 'return name.replace(b"Wiliam", b"William")'
would result in the following function being called:
def name_callback(name): return name.replace(b"Wiliam", b"William")
The email callback is quite similar:
git-filter-repo --email-callback 'return email.replace(b".cm", b".com")'
The refname callback is also similar, but note that the refname passed in and returned are expected to be fully qualified (e.g. b"refs/heads/master" instead of just b"master" and b"refs/tags/v1.0.7" instead of b"1.0.7"):
git-filter-repo --refname-callback ' # Change e.g. refs/heads/master to refs/heads/prefix-master rdir,rpath = os.path.split(refname) return rdir + b"/prefix-" + rpath'
The message callback is quite similar to the previous three callbacks, though it operates on a bytestring that is likely more than one line:
git-filter-repo --message-callback '
if b"Signed-off-by:" not in message:
message += b"\nSigned-off-by: Me My <self@and.eye>"
return re.sub(b"[Ee]-?[Mm][Aa][Ii][Ll]", b"email", message)'
The filename callback is slightly more interesting. Returning None means the file should be removed from all commits, returning the filename unmodified marks the file to be kept, and returning a different name means the file should be renamed. An example:
git-filter-repo --filename-callback '
if b"/src/" in filename:
# Remove all files with a directory named "src" in their path
# (except when "src" appears at the toplevel).
return None
elif filename.startswith(b"tools/"):
# Rename tools/ -> scripts/misc/
return b"scripts/misc/" + filename[6:]
else:
# Keep the filename and do not rename it
return filename
'
In contrast, the blob, reset, tag, and commit callbacks are not expected to return a value, but are instead expected to modify the object passed in. Major fields for these objects are (subject to API backward compatibility caveats mentioned previously):
An example of each:
git filter-repo --blob-callback '
if len(blob.data) > 25:
# Mark this blob for removal from all commits
blob.skip()
else:
blob.data = blob.data.replace(b"Hello", b"Goodbye")
'
git filter-repo --reset-callback 'reset.ref = reset.ref.replace(b"master", b"dev")'
git filter-repo --tag-callback '
if tag.tagger_name == b"Jim Williams":
# Omit this tag
tag.skip()
else:
tag.message = tag.message + b"\n\nTag of %s by %s on %s" % (tag.ref, tag.tagger_email, tag.tagger_date)'
git filter-repo --commit-callback '
# Remove executable files with three 6s in their name (including
# from leading directories).
# Also, undo deletion of sources/foo/bar.txt (change types are
# either b"D" (deletion) or b"M" (add or modify); renames are
# handled by deleting the old file and adding a new one)
commit.file_changes = [
change for change in commit.file_changes
if not (change.mode == b"100755" and
change.filename.count(b"6") == 3) and
not (change.type == b"D" and
change.filename == b"sources/foo/bar.txt")]
# Mark all .sh files as executable; modes in git are always one of
# 100644 (normal file), 100755 (executable), 120000 (symlink), or
# 160000 (submodule)
for change in commit.file_changes:
if change.filename.endswith(b".sh"):
change.mode = b"100755"
'
INTERNALS
You probably don’t need to read this section unless you are just very curious or you are trying to do a very complex history rewrite.How filter-repo works
Roughly, filter-repo works by runninggit fast-export <options> | filter | git fast-import <options>
where filter-repo not only launches the whole pipeline but also serves as the filter in the middle. However, filter-repo does a few additional things on top in order to make it into a well-rounded filtering tool. A sequence that more accurately reflects what filter-repo runs is:
Some notes or exceptions on each of the above:
Information about these steps is printed out when --debug is passed to filter-repo. When doing a --partial history rewrite, steps 2, 3, 7, and 8 are unconditionally skipped, step 5 is skipped if --replace-refs is update-no-add, and just the nuke-unused-refs portion of step 5 is skipped if --replace-refs is something else.
Limitations
Inherited limitationsSince git filter-repo calls fast-export and fast-import to do a lot of the heavy lifting, it inherits limitations from those systems:
There are also some limitations due to the design of these systems:
Intrinsic limitations
Some types of filtering have limitations that would affect any tool attempting to perform them; the most any tool can do is attempt to notify the user when it detects an issue:
Issues specific to filter-repo
Comments on reversibility
Some people are interested in reversibility of of a rewrite; e.g. rewrite history, possibly add some commits, then unrewrite and get the original history back plus a few new "unrewritten" commits. Obviously this is impossible if your rewrite involves throwing away information (e.g. filtering out files or replacing several different strings with ***REMOVED***), but may be possible with some rewrites. filter-repo is likely to be a poor fit for this type of workflow for a few reasons:
SEE ALSO
git-rebase(1), git-filter-branch(1)GIT
Part of the git(1) suiteNOTES
- 1.
- GitLab’s excellent docs on reducing repository size
- 2.
- GitHub’s otherwise dangerously out-of-date docs on removing sensitive data
| 06/07/2021 | Git 2.32.0.dirty |
Visit the GSP FreeBSD Man Page Interface.
Output converted with ManDoc.