Problem scenario
You have a massive amount of data to process. The amount is too big for the server you have. How do you write a program or algorithm to handle a very large set of data?
Possible Solution #1
Logically divide the memory by writing portions of the data to files. This is called “chunking” the data. Serially/sequentially process batches. Use external sorting with subfiles in parallel with multiple hard disks if the data set is very large.
…
Continue reading “What Do You Do if the Data Input Does Not Fit in Memory in Computer Programming?”