ENGAGE-290: Import a good, fresh snapshot of all the McCord data for the pilot

Metadata

Source
ENGAGE-290
Type
Sub-task
Priority
Major
Status
Closed
Resolution
Fixed
Assignee
N/A
Reporter
Michelle D'Souza
Created
2010-01-25T13:03:14.000-0500
Updated
2014-03-03T14:14:25.141-0500
Versions
N/A
Fixed Versions
  1. 0.3b
Component
N/A

Description

For the McCord pilot we require a complete set of data in both English and French. We need to get this data from McCord, import it into Couch, and test that it has imported correctly.

Attachments

Comments

  • y z commented 2010-02-01T13:29:43.000-0500

    This is a working import tool for the mccord data, however major refactoring and cleanup needed.

  • y z commented 2010-02-01T15:21:40.000-0500

    Updated Import Utility, ready for cleanup.

  • y z commented 2010-02-01T15:24:55.000-0500

    Too much functionality in 'main'. This code should be factored differently. At minimum it should be put into a method in the utilities file but may actually be better factored in a different class.

    line 25
    a hardcoded config file directory

    • change to using a relative path to the current directory

    line 40

    empty response is hardcoded - a change to what an empty response returned would cause this to break

    config file contains el paths to data block that are in HTML

    • we should document this or change the structure of the config file to be more intuitive

    prepareImport should use a 'converter' to externalize the actual conversion so that CouchDbUtility can be used for CSV, XML etc. This means adding to the XMLConverter an API which uses an URL

    line 67 and 71
    hardcoded URLs to the McCord data sources

    line 79 and 80
    hardcoded URLs to the database

    line 84
    hardcoded ids to exhibitions

    line 90 and 98
    hardcoded config file name

    line 92
    change the strategy of getting to the accessnumber to use an EL path

    line 107
    change the error handling to use logging

  • Michelle D'Souza commented 2010-02-02T10:17:05.000-0500

    mccord-pilot

  • y z commented 2010-02-03T16:11:46.000-0500

    Imported the data from McCord exhibition and artifact feeds. Still need to fix the encoding problem.

  • y z commented 2010-02-04T13:10:21.000-0500

    Final working importer with correct encoding.