Fixing MAX_PATH issues in GitLab

This post is a short reminder for my future self, for when I next have to fight with MAX_PATH issues and GitLab.

Yes, it's 2024, and I'm still being sabotaged by Window's archaic MAX_PATH restriction.

Window's maximum path length limitation.

In the much of the Windows API, the maximum length for a path is MAX_PATH, which is defined as 260 characters. In this day and age that is uncomfortably small, and so in Windows 10, version 1607 most MAX_PATH restrictions were removed from file and directory functions.

However, Microsoft are dedicated to backwards compatibility, almost to a fault. So even though the MAX_PATH restriction doesn't need to be there any more, it's still enforced, for back-compat reasons. If you want to create paths with more than 260 characters you'll need to set a value in the registry HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\FileSystem LongPathsEnabled (Type: REG_DWORD) to 1. You can do this using powershell, for example:

New-ItemProperty -Path "HKLM:\SYSTEM\CurrentControlSet\Control\FileSystem" -Name "LongPathsEnabled" -Value 1 -PropertyType DWORD -Force

or it can be enabled through group policy, with the policy Enable Win32 long paths.

Of course, even if you do that, you still can't open and edit paths using Windows File Explorer. 8 years after the limitation was removed. That would be too much to ask. Yes, I'm feeling snarky.

Git's long-path support

I ran into a MAX_PATH error that was happening in our GitLab builds at work recently, after a repository was moved to a different group. Doing so added an additional prefix to the checkout path, and suddenly 💥, GitLab could no longer check-out the repository:

error: unable to create file tracer/test/snapshots/
Security.AspNetCore5AutoUserEvents.SecurityOff.
extendedmode.TestUserLoginEvent_eventName=login.auto.
failure_bodyString=Input.UserName=NoSuchUser&
Input.Password=test.verified.txt: Filename too long

Now, granted, that filename kind of is too long, but that's beside the point 😅

Luckily Git does have support for bypassing the MAX_PATH Windows issue. You can enable support for long paths by setting the core.longpaths setting to true. For example:

git config --global core.longpaths true

Interestingly this works regardless of whether you've set the LongPathsEnabled registry key from the previous section. How can that be?

You can see the explanation for how it works in the commit that introduced the core.longpaths setting:

Many Windows wide char APIs support longer than MAX_PATH paths through the file namespace prefix (\\?\ or \\?\UNC\) followed by an absolute path…(We) introduce a function to optionally expand long paths using the \\?\ file namespace prefix…Contrary to MSDN docs, the GetFullPathNameW function doesn't seem to be limited to MAX_PATH, so we can use it to do the heavy lifting of the conversion (translate / to \, eliminate . and .., and make an absolute path).

So Git checks the length of any paths when it needs to perform a file operation. If the path is short, it just uses the standard APIs. If the path exceeds the MAX_PATH limitation and the core.longpaths setting is enabled then it converts the (potentially relative path) to an absolute path using GetFullPathNameW() and prepends \\?\. The resulting path can then be used to interact with the Windows APIs. Magic!

The really nice part of this is that it means you don't need to have admin access to the machine to checkout a git repo that has long paths. You can simply add the setting and bypass the issue.

Running scripts before git clone in GitLab

Which brings us to the crux of the solution: how to add that setting to a GitLab runner. If you have access to the configuration of the runner itself, that's no problem, you can just add the setting in the runner and your job is done.

However, if you don't have access to the runner, you'll need to make sure you add the setting before your repository is cloned.

My first thought was to add the setting in the before_script section:

build:
  stage: build
  tags: ["windowsversion:1809"]
+ before_script:
+   - git config --system core.longpaths true
  script:
    - if (Test-Path build-out) { remove-item -recurse -force build-out }

Unfortunately, that doesn't work. before_script runs before the job script executes (obviously) but after the repository has been cloned and the artifacts have been restored.

After a bit more googling, I came across the pre_clone_script option: the name sounds pretty promising! Unfortunately, this is configuration that goes in your runner configuration, not in your .gitlab-ci.yml. But there is a way to modify it—if you add a build variable to your .gitlab-ci.yml called CI_PRE_CLONE_SCRIPT you can use that to "pass through" to the runner. But that only appears to be on GitLab's SaaS runners, and it's deprecated in version 16. sigh.

Finally I came to the all important answer hooks:pre_get_sources_script. This hook is exactly what we need:

Use hooks:pre_get_sources_script to specify a list of commands to execute on the runner before cloning the Git repository and any submodules.

Adjusting our initial attempt to use hooks:pre_get_sources_script looks something like this:

build:
  stage: build
  tags: ["windowsversion:1809"]
+ hooks:
+   pre_get_sources_script
+     - git config --system core.longpaths true
  script:
    - if (Test-Path build-out) { remove-item -recurse -force build-out }

And voila, we have success!

Cloning into 'dd-trace-dotnet'...
remote: Enumerating objects: 168336, done.
remote: Counting objects: 100% (5289/5289), done.
remote: Compressing objects: 100% (2112/2112), done.
remote: Total 168336 (delta 4074), reused 4071 (delta 3152), pack-reused 163047 (from 1)
Receiving objects: 100% (168336/168336), 133.06 MiB | 24.91 MiB/s, done.
Resolving deltas: 100% (135723/135723), done.
Updating files: 100% (11458/11458), done.

Summary

In this short post I gave a brief description of the MAX_PATH limitation in Windows that restricts valid path lengths to 260 characters and showed how you can add support for long paths in Windows by setting a registry key. Git has its own long path support that can be enabled independently by setting core.longpaths=true, by running git config --global core.longpaths true for example.

If you're using GitLab, you need to make sure this setting is updated before running Git operations such as cloning the repository. You should use the hooks:pre_get_sources_script for this in GitLab 16+.