Debugging Production: When Git Submodules Fail Silently

The Mystery: Code Pushed, Production Unchanged

Picture this: You push a commit to GitHub. The push succeeds. You refresh your production site. Nothing changed.

You check again. Still old code. You clear your browser cache. Still old code. You wait 10 minutes. Still old code.

This is every developer's nightmare—the silent failure. No error message, no warning, just... nothing happens.

The Investigation Begins

Step 1: Verify Git History

$ git log --oneline -5
e92cb6f (HEAD -> master, origin/master) Fix production build
c2bad2d Revert unnecessary wrangler.toml build command
9049c68 Add pages_build_command to ensure build runs
2518aeb Remove all demo references
97a6119 Add type=button to prevent form submission

✅ Commits are there. Code is pushed. origin/master matches local.

Step 2: Check Cloudflare Pages Dashboard

Cloudflare Pages showed "Deployment successful" but the build output was using old code. The timestamp was recent, but the content was stale.

🤔 Something was wrong with the source being built, not the build process itself.

Step 3: Inspect Git Status

$ git status
On branch master
Your branch is up to date with 'origin/master'.

nothing to commit, working tree clean

Clean working tree. No uncommitted changes. Everything looks normal.

Step 4: The Smoking Gun

$ git ls-tree HEAD | grep clodo-starter-template
160000 commit 1c53a56d... clodo-starter-template

🚨 There it is. Mode 160000 means git is tracking clodo-starter-template as a submodule commit reference, not as actual directory contents.

Understanding Git Submodules

What Are Git Submodules?

Git submodules allow you to include one git repository inside another. They're commonly used for:

• Embedding third-party libraries
• Sharing code between projects
• Managing mono-repos

How They Work (When They Work)

# Properly configured submodule
$ cat .gitmodules
[submodule "clodo-starter-template"]
    path = clodo-starter-template
    url = https://github.com/tamylaa/clodo-starter-template.git

# Git tracks a specific commit
$ git submodule status
1c53a56d... clodo-starter-template (heads/master)

When properly configured:

1. Parent repo references a specific commit in the submodule
2. .gitmodules file defines the submodule location
3. git submodule init and git submodule update fetch the code

Our Problem: The Half-Configured Submodule

Our situation was worse—we had a submodule reference without the configuration:

$ cat .gitmodules
cat: .gitmodules: No such file or directory

$ git ls-tree HEAD clodo-starter-template
160000 commit 1c53a56d... clodo-starter-template

$ ls -la | grep clodo-starter-template
drwxr-xr-x  clodo-starter-template/

This created a nightmare scenario:

• ✅ Locally: Directory exists with full contents
• ❌ In git: Only a commit reference is tracked
• ❌ On Cloudflare Pages: Build fails to fetch submodule (no .gitmodules)
• ❌ Build logs: No clear error message

How Did This Happen?

The most common causes:

Scenario 1: Accidental git add

# You accidentally add a directory that's already a git repo
$ cd project
$ git clone https://github.com/example/library.git
$ git add library/  # ⚠️ git sees .git folder, treats as submodule
$ git commit -m "Add library"

Scenario 2: Manual .gitmodules Deletion

# Someone removes .gitmodules but not the submodule reference
$ rm .gitmodules
$ git add .gitmodules
$ git commit -m "Remove submodules"
# ⚠️ Submodule reference still exists in git index!

Scenario 3: Incomplete Submodule Removal

# Wrong way to remove submodule
$ rm -rf clodo-starter-template/
$ git add .
$ git commit
# ⚠️ Reference remains, directory is gone

The Fix: Complete Submodule Removal

Step 1: Remove from Git Index

$ git rm --cached clodo-starter-template
rm 'clodo-starter-template'

The --cached flag removes from git tracking but keeps local files intact.

Step 2: Prevent Future Issues

$ echo "clodo-starter-template/" >> .gitignore
$ git add .gitignore

Now git will ignore the directory completely.

Step 3: Commit and Push

$ git commit -m "Fix: remove broken submodule reference"
[master e92cb6f] Fix: remove broken submodule reference
 1 file changed, 1 insertion(+)
 1 file changed, 0 insertions(+), 0 deletions(-)
 delete mode 160000 clodo-starter-template

$ git push origin master

Step 4: Verify the Fix

$ git ls-tree HEAD | grep clodo-starter-template
# (no output - submodule reference removed!)

$ git status
On branch master
nothing to commit, working tree clean

✅ Submodule reference gone. Directory now properly ignored.

Why Cloudflare Pages Failed Silently

Cloudflare Pages build process:

1. Clone repository - Gets commit references including submodule pointer
2. Check for .gitmodules - Not found!
3. Skip submodule init - Can't fetch without configuration
4. Continue build - Directory doesn't exist, but build doesn't fail immediately
5. Use cached build - Falls back to last successful build

The result: New commits are "deployed" but old code is served.

Why No Error Message?

The build system didn't consider missing submodule initialization a build failure—just a skipped step. The actual build command (npm run build) succeeded because it didn't reference the missing directory.

Lessons Learned: Git Submodule Best Practices

1. Avoid Submodules When Possible

Submodules add complexity. Consider alternatives:

• npm/yarn packages for JavaScript dependencies
• Separate repositories with CI/CD integration
• Mono-repos with tools like Nx or Turborepo
• Git subtrees if you need embedded repos

2. If You Must Use Submodules...

Proper initialization:

# Add submodule correctly
$ git submodule add https://github.com/user/repo.git path/to/submodule

# Always commit .gitmodules
$ git add .gitmodules path/to/submodule
$ git commit -m "Add submodule: repo"

# Clone repository with submodules
$ git clone --recursive https://github.com/user/parent-repo.git

# Update submodules in existing clone
$ git submodule init
$ git submodule update

3. Remove Submodules Completely

The correct removal process:

# 1. Remove submodule entry from .gitmodules
$ git config -f .gitmodules --remove-section submodule.path/to/submodule

# 2. Remove submodule entry from .git/config
$ git config -f .git/config --remove-section submodule.path/to/submodule

# 3. Remove from index
$ git rm --cached path/to/submodule

# 4. Remove .git metadata
$ rm -rf .git/modules/path/to/submodule

# 5. Remove files
$ rm -rf path/to/submodule

# 6. Commit
$ git commit -m "Remove submodule: path/to/submodule"

4. Configure Build Systems for Submodules

Most CI/CD platforms need explicit configuration:

GitHub Actions:

- name: Checkout code
  uses: actions/checkout@v3
  with:
    submodules: recursive  # ← Important!

Cloudflare Pages:

# wrangler.toml
[build]
command = "git submodule update --init --recursive && npm run build"

Vercel: Auto-detects and initializes submodules by default.

Alternative: Our Chosen Solution

Instead of fighting with submodules, we chose a cleaner approach:

Separate Repositories

• clodo-dev-site - Marketing website
• clodo-starter-template - StackBlitz demo (separate repo)
• clodo-framework - Framework npm package

Benefits

1. Independent versioning - Update demo without touching website
2. No build dependencies - Each deploys independently
3. Clear separation - Demo code isn't mixed with marketing site
4. GitHub template - Can mark demo as template for easy forking
5. StackBlitz integration - Direct URL to separate repo

Trade-offs

• ❌ Manual sync if demo needs website changes
• ❌ Can't test demo changes locally within website repo
• ✅ But: Demo is deployed to StackBlitz, not our site
• ✅ Simpler mental model and deployment

Debugging Checklist for Silent Build Failures

When production doesn't update after pushing code:

1. Verify Git State

# Check commit history
$ git log --oneline -10

# Verify remote sync
$ git fetch
$ git status

# Check for uncommitted changes
$ git diff
$ git diff --staged

2. Check for Submodule Issues

# List submodule references
$ git ls-tree HEAD | grep "^160000"

# Check .gitmodules
$ cat .gitmodules

# Submodule status
$ git submodule status

3. Inspect Build Logs

• Cloudflare Pages: Dashboard → Deployments → View build log
• Vercel: Deployment → View function logs
• Netlify: Deploys → Deploy log

4. Test Build Locally

# Run production build locally
$ npm run build

# Simulate fresh clone
$ cd /tmp
$ git clone https://github.com/user/repo.git
$ cd repo
$ npm install
$ npm run build

5. Check Cache

Many platforms cache builds. Try:

• Force rebuild (Cloudflare: "Retry deployment")
• Clear cache (Vercel: "Clear cache and redeploy")
• Push empty commit to trigger rebuild

Tools for Detection

Git Hooks

Prevent accidental submodule commits:

#!/bin/bash
# .git/hooks/pre-commit

# Check for accidental submodule additions
SUBMODULES=$(git diff --cached --name-only --diff-filter=A | \
    while read file; do
        if [ -d "$file" ] && [ -d "$file/.git" ]; then
            echo "$file"
        fi
    done)

if [ -n "$SUBMODULES" ]; then
    echo "Error: Attempting to add directories with .git folders:"
    echo "$SUBMODULES"
    echo "Did you mean to add these as submodules?"
    exit 1
fi

CI/CD Validation

Check for submodule misconfigurations:

# .github/workflows/validate.yml
name: Validate Repository
on: [push, pull_request]

jobs:
  check-submodules:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      
      - name: Check for broken submodules
        run: |
          # Find submodule references
          REFS=$(git ls-tree HEAD | grep "^160000" | cut -f2)
          
          if [ -n "$REFS" ]; then
            echo "Found submodule references:"
            echo "$REFS"
            
            # Verify .gitmodules exists
            if [ ! -f .gitmodules ]; then
              echo "Error: Submodules found but .gitmodules missing!"
              exit 1
            fi
          fi

Conclusion

Silent failures are the worst kind of bugs. They don't crash, they don't error—they just quietly break your deployment pipeline.

The broken git submodule issue taught us:

1. Verify the source - Don't assume pushed code is deployed code
2. Submodules need full configuration - Or avoid them entirely
3. Test fresh clones - Your local environment lies
4. Check build logs - Even "successful" builds can hide issues
5. Simplify when possible - Separate repos beat complex submodule hierarchies

By removing the broken submodule and moving to separate repositories, we eliminated an entire class of deployment issues. Sometimes the best debugging is prevention.

Related Reading

• Building an Instant "Try Now" Experience - The full integration story
• The Cloudflare Infrastructure Myth - Understanding Cloudflare's architecture
• How Instant Try-It Demos Boost Developer Adoption - Measuring the impact