Data Migration

Data migration for any implementation can range from being relatively straight-forward to highly complex.

Our standard implementation pricing includes the work to be undertaken on the “PMWeb side”. This is the configuration of PMWeb fields such that they align with and support the required process flows; and align with the data to be imported – together with the importation of that data and verification of its correctness post-importation. Our pricing also includes advice to the Client on the activities being undertaken “Client-side” to prepare the data for migration.

In some implementations we are asked to assist in the Client-side work of data preparation. In these circumstances, we recommended a two-stage approach. Stage 1 is a fixed price to conduct a scoping study; with Stage 2 work to be determined based on the results of that study. Stage 1 involves review of the locations of existing data, the condition of that data and the approach that most suits the Client. During Stage 1 we would do high level data mapping, create a data migration inventory / plan; and perform some data profiling (i.e. existence of field, nullability, keys). At the end of Stage 1, we would supply the following Deliverables:

• Data Migration Plan that will contain a full list of the data artefacts requiring migration by release
• High Level Data Mapping between the Source and Target
• Data Profiling results from source data (which will form part of the data mapping).

We recognize that ultimately a successful migration is not that which has the best code or best technology, but rather the one with the best planning and data analysis.

The high-level methodology that would apply in a series of Sprints that culminate in a Go Live for organisational Release (i.e. with all data migration complete) is as follows:

Such migration delivery is usually achieved through a structured approach that recognizes the Sprint-based system delivery approach:

Initial considerations

Supplier data migration is typically a straightforward process. Migrating Contract and Project Data, however, can be less straightforward. There are multiple considerations for the Client to take into account in its process of preparing a clean set of data and documents for migration to PMWeb. These include:

• Original Contract – there are seldom issues in migrating original contract information.
• Variations – issues can include sequencing when the variations came through, what period, cost codes, etc. The file information needs to include the core fields, vendor ID, cost period, date, cost codes, amounts. Typically these also need to be linked to the invoices, which can take time to properly compile. Variations can be “plus or minus”. Depending on when item was added/removed determines when you can or cannot bill against it. How are exceptions to be handled across many hundreds of records? In principle, this is relatively straightforward if all data is clean and fields correctly aligned; but, if not, data integrity issues can occur
• Invoices – individual transactions for the invoices can also become tricky because you then have to start capturing which variations came in (and when) and verify that they actually exist - so linking an invoice to a variation and synchronizing the periods will work well as long as there is clean data. If the data is not clean, it can end up being like finding a needle in a haystack to find problems.

There are several methods we follow to ensure a successful migration of the Client’s data into the new system. Our main goal will be to keep the data migration process as simple as possible and only move data that is absolutely necessary. We have a several techniques that can be used in this situation – such as creating baseline records. This would be subject to the Client specific requirements, with the Client undertaking the necessary work to prepare the data for migration in the appropriate format. The following are details of the approach we adopt if undertaking some or all of the Client-side work and may be of use to the Client in its preparations:

Baseline Records

For lengthy or complex projects and their associated contracts, we create a set of baseline records that catch us up to current status with all the associated records, typically including variations and invoices. For example, one base record, one base entry for Variations that captures all the current variations in a single record, one base entry for Invoices that catches up to the current state of the Invoicing. Then, moving forward, individual transactions are entered directly or imported via the Migration process.

Contract status is also a consideration:

• Projects and Contracts about to commence: Ccommencing projects and contracts should be considered as UAT test cases. This not only validates business rules, but ensures relevant meta and system data is converted / present to accommodate upcoming projects and contracts
• In flight projects and contracts – about to close: Where possible, a focus is given to finalizing as many elements as possible. Of critical importance here is minimizing the “change freeze” period such that business operations are not impacted and data remains consistent 
• In flight projects and contracts – long term ones: As above, however a greater focus is given to data quality in the source system and ensuring that the source system has the appropriate accuracy. Longer running projects and contracts can lose a level of focus on data quality
• Closed/archived contracts/projects: Closed and archived contracts/projects are migrated based on a set of flexible decision criteria that essentially balances the trade-off between information value and its potential to pollute new data. Pollution occurs due to a range of factors including changed business rules, business terms and artefact codes such as account codes, asset codes, location boundaries etc.

We also find that there is a tight relationship between testing and data migration / conversion. The data quality rules that apply to various transaction (contract/project) states are often refined during user tests as system familiarity increases. As such, the migration approach is typically designed for flexibility and close alignment, so that what information is migrated (and how) has ongoing agility until the go-live date.

Data Quality Approach

Typically data from capital works project and contract records is different to most generic application data. There is a blending of data from various other systems / sources. The data may be accurate in context to its source system, but when blended with other data its “Quality” may change.

What constitutes a complete transaction that can be migrated can be complex and have different data owners with different business requirements for data. The data quality approach will define what data is required and what level of quality is required for specific items. Achieving the right result migration result is a foundation for successful operation.

Transition considerations

The transition approach will be developed in consultation with the Client once more is known about the locations and relative quality of data. Typical transition considerations include:

• Mapping the data.  Different systems will treat data differently.  Often there will not be a direct map between the systems.  This is especially true because during the configuration process you are still determining where destination data should go.  This is why we propose a pilot migration running parallel processes.  Also important in this process is to determine what data actually needs to come over, so we don’t spend time mapping fields and finding a place for information that doesn’t really matter in the end
• Determining the Data Quality approach. Project and contract data in an organisation such as the Client can include data from a range of sources across capital works and non-capital works business activities. Migration can involve the blending of data from various other systems / sources. Data may be accurate in context to its source system, but when blended with other data its “Quality” may change. What constitutes a complete transaction that can be migrated can be complex and have different data owners with different business requirements for data. The data quality approach defines what data is required and what level of quality is required for specific items. Quality metrics typically include, age, value, completeness, correctness & timeliness. Consider iterative Improvement of data quality until a target system is loaded with agreed quality targets. Possibly also consider improving legacy data where appropriate (i.e. manual correction)
• Sign-off. The Client will need to identify the person or people authorised to sign-off that (a) the data has been cleansed and is ready for migration; and (b) the data has been correctly migrated. Migration sign-off will prove Lineage, Transformation and Reconciliation; and confirm Data Ownership
• Sensitive data. Data loaded progressively in parallel with configuration should not include sensitive data to avoid the need for masking for testing phases
• Cloud-based hosting. Should the Client decide to have PMWeb Cloud-hosted, extract the data at the Client and load in the cloud (i.e. clear lines of separation due to security implications of directly connecting from cloud)
• Determining when to cut over.  With a large organization and many users it is usually not feasible to turn one system off; turn the other one on; and train all the users at once so they can start using the new system, as this would require a large amount of dedicated resources.   

It is important that these typical data migration issues are satisfactorily addressed in advance to ensure a smooth transition.

Transition process

The developed Data Migration Plan will further detail the processes agreed with the Client based on the overall approach. PMWeb has a number of options for integrating data from existing systems.  The method used is determined by factors such as the type of data being integrated; the frequency of data exchange; and the direction of the data exchange.  See Integration page.

Our data migration experience includes:

• SAP
• Oracle Financials
• Oracle Primavera P6
• Oracle Primavera Contract Manager (PCM) / Expedition
• Oracle Unifier
• Procore
• Prolog Manager
• Autodesk Constructware
• Timberline
• ViewPoint
• DB2
• ABB Ellipse
• eCMS
• Epicor
• AS400
• SharePoint
• HP Desksite
• eBuilder