Problem scenario
You run a grep command but it fails. You get a message about "memory exhausted." What should you do?
Solution
Possible Solution #1
If you are comparing two files line-by-line, try reversing the order of the file arguments for the grep command. For example try these two versions:
grep -f a.txt b.txt
grep -f b.txt a.txt
If the file in the left argument is smaller, we know it uses less memory.
Possible Solution #2
If you have enough disk space, add swap space. See this posting if you need more assistance.
Possible Solution #3
See How Do You Add Memory to a Server? for resizing the VM to add more memory.
@echo off
setlocal enabledelayedexpansion
set file1=”C:\path\to\file1.txt”
set file2=”C:\path\to\file2.txt”
set batch_size=1000
rem Split and compare the files incrementally
for /l %%a in (1,1,100000) do (
set /a start_line=(%%a-1)*batch_size+1
set /a end_line=%%a*batch_size
if !start_line! gtr 100000 goto :done
if !end_line! gtr 100000 set /a end_line=100000
set outfile=”diff_batch_%%a.txt”
head -n !end_line! %file1% | tail -n +%start_line% | grep -Fxvf %file2% > !outfile!
echo Differences in lines !start_line! to !end_line!:
type !outfile!
)
:done
endlocal