Our system used to only receive input data as large ASCII flat files. These were generated on a daily basis by some mainframe computers. The loading process on our system parsed the files according to a record layout. The result was inserted into an Oracle database which our applications query.Last year we started receiving some files in XML format. There was a big concern about the size of such files. The old input files were minimal. You just got the data arranged in a specified format. Many worried the overhead of XML tags would make the files too large to handle. So far we have not received too much data in these files. The technique has not been volume tested.
Now we are preparing to receive a combination of XML and ASCII text files next year. There will be format changes for both of these files. The ASCII text files just have a different record layout. The XML files will be formatted according to a new schema. I suspect in the future that we will only receive files in XML format. Then we will only have to deal with XML schema changes. The real thing to look out for is the growth of XML input file size. We do have a plan to split the XML files into small chunks that we can reasonably process.
