# Idea

Download complete single html pages, to make them available (along with the respective assets) for 'offline' viewing when the old website is no longer available.

# Requirements/Expectations

The entry point is a manual list of URLs from the old website, with a target 'Navigation' term (in the new system)
The expected data structure is like the following:

-----------------------------------------------------
| Page Title | URL | Top-Parent | Parent1 | Parent2 |
-----------------------------------------------------

Specifically, for each column:

'Page Title': This is the page title, can be ignored as the offline HTML page is expected to have the title attribute. Still keeping it for reference.
'URL': This is the unique identifier of the page, should be the fully expanded, external path (no internal paths like node/1234).
'Top-Parent', 'Parent1', 'Parent2': These are all the names of terms in the target 'Navigation' taxonomy, they may not all be present, some examples of expected values are
      * Top-Parent e.g. 'Branche'
      * Top-Parent>>Parent1 e.g. 'Branche'>>'Rahmenbedingungen'
      * Top-Parent>>Parent1>>Parent2 e.g. 'Branche'>>'Rahmenbedingungen'>>'Abkommen'


# Folder/File structure, names and expected location

source_urls.csv: This one is delivered/exported, should not be edited further.
Fields must be tab delimited.
Text no delimited.

content: This is the target folder for storing offline versions of the legacy pages.

scripts/crawl.bash: This is a wrapper around wget; reads the URLs from the second column of the source_urls.csv file and stores them into the content folder. This should be run like `cd DRUPAL_ROOT && bash ../migration/scripts/crawl.bash`

Run `cd scripts && composer install` to install the dependencies for the import script.

scripts/import_nodes.php: This is the script that imports the URLs into the database. It does one thing; creates a new node, if none is found with the incoming URL. This should be run like `cd DRUPAL_ROOT && drush scr ../migration/scripts/import_nodes.php 2> stderr.log`

docroot/_legacy: This (soft-linked) folder should point to the content folder, so that legacy pages are avainable e.g. http://svv.dev/_legacy/www.svv.ch/de/node/4079.html. This could be done by running `cd DRUPAL_ROOT && ln -s ../migration/content _legacy`


# Iterations/Corrections to the source csv list

After updating the source list (source_urls.csv), the import script (scripts/import_nodes.php) can be run again. It will then update existing nodes (using the URL to locate them) allowing bulk updates to the 'Title' and 'Navigation' fields.


# Errors in navigation paths

When running the import_nodes.php script, the STDERR will contain - if needed - wrongs Navigation paths information, for example:

      Wrong Navigation path for URL:[http://www.svv.ch/de/node/4179], input path [Branche>>Versicherungszweige>>Sachversicherung]