Creating 32,000 Web Pages in About 30 Minutes
This is a story of how 32,000 web pages were created in about 30 minutes. The year was 2004, the internet was still young, Google was still new and web designers were hard to find. I had about 40 years experience as a programmer and knew some Html from running a blog online for about 4 years.
USMap
The client had established their domain and a few web pages and was ready to setup pages for cities within all 50 states with 1,500 or more residents. I came into the picture as a programming consultant. The client provided a model "state" and "city" web page and 50 "DOC" files (one per state) that contained all of the zipcode/city_names per each state.
south_carolina
Each city page would need two major inserted blocks of web links and various customizations where the state name and city names were to be inserted repeatedly. So three boilerplate files were created Part_1 for the header and early lines of Html code. The part_2 file held the middle portion and part_3 held the footer.
I used the Linux system to do this project because of the tools available and because of the ability of one program or script to feed data (pipe) into another program or script. The major tools I used were Awk and Sed.
The work was divided up into three major parts Makeblock1, Makeblock2 and Makeflp. The major role of the first two scripts (shown below) was to setup the 2 letter state codes (Makeblock1) and city names (Makeblock2) to be processed. All the work is done in the third script Makeflp.
Makeblock1
Makeblock2
This third script (Makeflp, below) calls Awk to build a file of hyperlinks (href) and to build a script to perform the final customization and assembly, Makepages. Following the Awk call, a temp file is created with part_2, href and part_3, naming it part_2b. The last step is the execution of the script Makepages. This "run time" script calls Sed to edit replace boilerplate "AAA", "BBB" & "CCC" with city name, 2 letter state code and state name. In addition it appends part_2b to part_1 storing the result in a file named for the city.
Makeflp
Here is a partial list of city files in the state directory for Rhode Island.
ExampleRI
