Data Assessment
Data Assessment
                                                          loaded to Gold
                                                          Errors
                                                          In progress
                                                          Total
                    10
                     8
                     2
                    20
Azure_Storage/ContainerMenuBlade/~/overview/storageAccountId/%2Fsubscriptions%2F34096212-9213-4f72-b8cb-967fdce09a76%2Fre
          WinWire Comments
          able to parse schema-> prc_meetings_list
          able to parse schema-> prc_project_permission_manifest_list
AccountId/%2Fsubscriptions%2F34096212-9213-4f72-b8cb-967fdce09a76%2FresourceGroups%2Frg-mdap-dev%2Fproviders%2FMicrosoft
          Error category          Metadata created   delta tables created   bronze loaded
                                  Y                  Y
                                  Y                  Y
mdap-dev%2Fproviders%2FMicrosoft.Storage%2FstorageAccounts%2Fdlsmdapdev/path/fsmdapdev/etag/%220x8DB6C0E7D26261E%22/
          silver and gold loaded   Sprint #     Is Full Load Done?   Is Incremental Load Done?
                                              6 No                   No
                                              6 No                   No
mdapdev/etag/%220x8DB6C0E7D26261E%22/defaultEncryptionScope/%24account-encryption-key/denyEncryptionScopeOverride~/false
           Is WW Validation Done?     Is WT Validation Done?
           YTS                        YTS
           YTS                        YTS
tion-key/denyEncryptionScopeOverride~/false/defaultId//publicAccessVal/None
Table_Name                         Column_Name
MEETINGS                           meetings
PROJECT_PERMISSION_MANIFEST_LIST   tools
RFIS                               questions
SUBMITTALS                         approvers
DOCUMENTS                          created_by
RFIS                               assignees
RFIs                               created_by
RFIS                               rfi_manager
RFIS                               responsible_contractor
SUBMITTALS                         submittal_package
USERS                              vendor
SUBMITTALS                         ball_in_court
USERS                              address
Connected Dataset (Toric)   Relationship Key
Tools [List] id
Users [List]                id
Users [List]                id
Users [List]                id
Users [List]                id
Users [List]                id
Vendors [List]              id
Users [List]                id
Vendors [List]              id
Users [List]                id
Notes
would likely need to broken into a child table
DOCUMENTS file 3N
DOCUMENTS created_by 2Y
MEETINGS                           meetings                         3Y
PROJECT_PERMISSION_MANIFEST_LIST   tools                            3Y
RFIS questions 3Y
RFIS                               assignees                        2Y
DRAWINGS                           current_revision                 2N
OBSERVATIONS                       origin                           2N
OBSERVATIONS                       assignee                         2N
RFIS responsible_contractor 2Y
SUBMITTALS approvers 3Y
SUBMITTALS submittal_package 2Y
SUBMITTALS ball_in_court 1Y
DOCUMENTS                          children                         1N
DRAWINGS                           thumbnail_url                    1N
DRAWINGS                           large_thumbnail_url              1N
FORMS                              thumbnail_url                    1N
FORMS                              created_by                       1N
FORMS                              attachment                       1N
FORMS                              fillable_pdf                     1N
INCIDENTS                          created_by                       1N
OBSERVATIONS                       created_by                       1N
OBSERVATIONS                       type                             1N
OBSERVATIONS                       location                         1N
PHOTOS                             uploader                         1N
PHOTOS                             links                            1N
RFIS                               cost_impact                      1N
RFIS                               priority                         1N
RFIS                               rfi_manager                      1N
RFIS                               schedule_impact                  1N
RFIS                               assignee                         1N
RFIS                               created_by                       1N
USERS                              vendor                           2Y
SUBMITTALS                         created_by                       1N
SUBMITTALS                         recieved_from                    1N
SUBMITTALS                         specification_section            1N
SUBMITTALS                         submittal_manager                1N
SUBMITTALS                         type                             1N
SUBMITTALS   responsible_contractor   1N
SUBMITTALS   status                   1N
USERS        address                  1Y
VENDOR       logo                     1N
VENDOR       project_ids              1Y
Comments                                             WW Comments
Doesn't contain arrays, a flattening procedure
could be implemented. Considering exclusion of
this type.
N/A id
N/A           id
N/A           id
N/A
N/A           id
N/A id
N/A id
N/A           id
23            N/A
            2
         1252 Full Load
            9
            1
         2684
      1696555
       190056
 3360839
  384638
     274
  872800
  139095
   21484
  830015
  135646
   77855
     174
       1
       0
 1056508
   85246
    1001
  849457
    7351
15878636
       0
  188909
 3664119 ICCF_ORASEQ
  136743
  113560
17341724
   15836
   19950
 1208500
 3634501
    3216
 1652432
    7482
  554799
   93719
      15
   99999
    1264
  263274
  172731
  677390
18215629
 1406619
 2191705
 1408847
15213810
  545960
  324256
    1500 Full Load
    1510
  325158
  467122
  126288
11336356
  598878
 1605598
     885
     314
       8
      39
81335920
    2550 Full Load
     416
 2724960
    1444
      38
     194
    2179 Full Load
   31615 Full Load
   31622 Full Load
      33 Full Load
   47058 Full Load
      51 Full Load
   22240 STATE_CODE
     393 ID
      38
     887
    4442
       9
  234896
 3077421
      54
    3688
     609
    7634
34148599
31088621
28987015
28977702
  780640
   15240
    3
   60
  210
14130
 7989
    3
15784
    5
    6
   21
  222
  612 Full Load
 2188
 3805 Full Load (primary key is: e_emp_no)
      Views
      Views
      Views
      Views
    7 Full Load (primary key is: PAYTYP_CODE)
Path
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
AP Rejects Tracking.xlsx
Jeff Numbers - Q3 2022.xls
Monthly Paydex.xlsx
https://whitingturner.sharepoint.com/:x:/r/sites/CMiCTeam2/Shared%20Documents/Analytics/Sharepoint%20ETL%20Files/C
https://whitingturner.sharepoint.com/:x:/r/sites/CMiCTeam2/Shared%20Documents/Analytics/Sharepoint%20ETL%20Files/C
https://whitingturner.sharepoint.com/:x:/r/sites/CMiCTeam2/Shared%20Documents/Analytics/Sharepoint%20ETL%20Files/C
https://whitingturner.sharepoint.com/:x:/r/sites/CMiCTeam2/Shared%20Documents/Analytics/Sharepoint%20ETL%20Files/R
Info Sys & Analytics - CR Solutions - WT Submissions - All Items (sharepoint.com)
https://whitingturner.sharepoint.com/sites/CMiCTeam2/Lists/EEO%20Liaisons/AllItems.aspx
Info Sys & Analytics - Job Access Audit List - All Items (sharepoint.com)
Info Sys & Analytics - LK-Departments (Active) - All Items (sharepoint.com)
Info Sys & Analytics - LK-WT Offices - All Items (sharepoint.com)
Info Sys & Analytics - New Power App/Flow Request List - My Requests (sharepoint.com)
Info Sys & Analytics - Plangrid App to Flow - All Items (sharepoint.com)
Info Sys & Analytics - Project Data Mapping - All Items (sharepoint.com)
Info Sys & Analytics - SEC-External User Job Security - All Items (sharepoint.com)
Info Sys & Analytics - SEC-Project Analytics Job Security - All Items (sharepoint.com)
Info Sys & Analytics - SEC-RDOFO by WT Office - All Items (sharepoint.com)
Info Sys & Analytics - SEC-Schedule Managers - All Items (sharepoint.com)
Info Sys & Analytics - SEC-SilkRoad College Mapping - All Items (sharepoint.com)
Info Sys & Analytics - SEC-Sr Supt by Department - All Items (sharepoint.com)
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
File Name / Folder Name   EndPoint
N/A                       N/A
N/A                       N/A
N/A                       N/A
N/A                       N/A
N/A                       N/A
N/A                       N/A
N/A                       N/A
N/A                       N/A
N/A                       N/A
N/A                       N/A
N/A                       N/A
N/A                       N/A
N/A                       N/A
N/A                       N/A
N/A                       N/A
N/A                       N/A
N/A                       N/A
N/A                       N/A
N/A                       N/A
N/A                       N/A
N/A                       N/A
N/A                       N/A
N/A                       N/A
N/A                       N/A
N/A                       N/A
N/A                       N/A
N/A                       N/A
N/A                       N/A
N/A                       N/A
N/A                       N/A
N/A                       N/A
N/A                       N/A
N/A                       N/A
N/A                       N/A
N/A                       N/A
N/A                       N/A
N/A                       N/A
N/A                       N/A
N/A                       N/A
N/A                       N/A
N/A                       N/A
N/A                       N/A
N/A                       N/A
N/A                       N/A
N/A                       N/A
N/A                       N/A
N/A                       N/A
N/A                                      N/A
N/A                                      N/A
N/A                                      N/A
N/A                                      N/A
N/A                                      N/A
N/A                                      N/A
N/A                                      N/A
N/A                                      N/A
N/A                                      N/A
N/A                                      N/A
N/A                                      N/A
N/A                                      N/A
N/A                                      N/A
N/A                                      N/A
N/A                                      N/A
N/A                                      N/A
N/A                                      N/A
N/A                                      N/A
N/A                                      N/A
N/A                                      N/A
N/A                                      N/A
N/A                                      N/A
N/A                                      N/A
N/A                                      N/A
N/A                                      N/A
N/A                                      N/A
N/A                                      N/A
N/A                                      N/A
N/A                                      N/A
N/A                                      N/A
N/A                                      N/A
N/A                                      N/A
N/A                                      N/A
N/A                                      N/A
AP Rejects Tracking.xlsx                 N/A
Jeff Numbers - Q3 2022.xls               N/A
Monthly Paydex.xlsx                      N/A
CR Solutions cleanup                     N/A
CR Solutions Daily Data as of 20240215   N/A
CR Solutions Daily Data as of 7-27-23    N/A
Recruiting Needs                         N/A
CR Solutions - WT Submissions            N/A
EEO Liaisons                             N/A
Job Access Audit Table                   N/A
LK-Departments (Active)                  N/A
LK-WT Offices                            N/A
New Power App/Flow Request List          N/A
Plangrid App to Flow                  N/A
Project Data Mapping                  N/A
SEC-External User Job Security        N/A
SEC-Project Analytics Job Security    N/A
SEC-RDOFO by WT Office                N/A
SEC-Schedule Managers                 N/A
SEC-SilkRoad College Mapping          N/A
SEC-Sr Supt by Department             N/A
ADMIN EQUIPMENT LIST                  ADMIN EQUIPMENT LIST
CHECKLIST INSPECTION SCHEDULES LIST   CHECKLIST INSPECTION SCHEDULES LIST
CHECKLIST INSPECTIONS SECTIONS LIST   CHECKLIST INSPECTIONS SECTIONS LIST
CHECKLIST SECTIONS                    CHECKLIST SECTIONS
CHECKLIST SECTIONS LIST               CHECKLIST SECTIONS LIST
CONSTRUCTION REPORT LOGS              CONSTRUCTION REPORT LOGS
FORMS LIST                            FORMS LIST
INCIDENTS                             INCIDENTS
OBSERVATIONS LIST                     OBSERVATIONS LIST
RFIS LIST                             RFIS LIST
MEETINGS LIST                         MEETINGS LIST
PROJECT PERMISSION MANIFEST LIST      PROJECT PERMISSION MANIFEST LIST
9 9
9 9
9 9
                         9                                   9
                         9                                   9
9 9
                         9                                   9
                         9                                   9
                         9                                   9
               JSGU__IU__UPDATE_DATE
               OMC__IU__UPDATE_DATE
               WRL__IU__UPDATE_DATE
               INV__IU__UPDATE_DATE
               JCGJ__IU__UPDATE_DATE
               OMCT__IU__UPDATE_DATE
               OPUDF__IU__UPDATE_DATE
               OMOPS__IU__UPDATE_DATE
               SDPICKR__IU__UPDATE_DATE
     9       9 VDIST__IU__UPDATE_USER
     2       3 N/A
     2       3 correction_date
     2       3 correction_date
     2       3 UPDATED
     2       3 UPDATED
     2       3 correction_date
     2       3 UPDATED
     2       3 UPDATED
     3       3 UPDATED
     2       3 UPDATED
     2       4 REFRESH_DATE
     2       3 UPDATED
     2       3 UPDATED
     2       3 UPDATED
     3       3 UPDATED
     2       3 TIMESTAMP
     2       3 UPDATED
     2       3 UPDATED
     3       3 UPDATED
     2       3 UPDATED
     3       3 created_at
     3       3 N/A
     6 N/A
NA     N/A
     6 N/A
     6 N/A
NA     N/A
     6 N/A
     6 N/A
     4 N/A
     4 N/A
     4 N/A
     4 N/A
     4 N/A
     4 N/A
4 N/A
4 N/A
4 N/A
4 N/A
4 N/A
4 N/A
4 N/A
4 N/A
5       5
5       5
5       5
5       5
5       5
6       6
5       5
5       5
5       5
5       5
6       6
6       6
7 7
7       7
7       7
7       7
7       7
7 7
7 7
7       7
7       7 UPDATED
Watermark Column               In Watermark Column is Null then
CLS__IU__UPDATE_DATE           CLS__IU__CREATE_DATE
DEPT__IU__UPDATE_DATE          DEPT__IU__CREATE_DATE
RESPONSIBLE__IU__UPDATE_DATE   RESPONSIBLE__IU__CREATE_DATE
INSTYP__IU__UPDATE_DATE        INSTYP__IU__CREATE_DATE
JCDT__IU__UPDATE_DATE          JCDT__IU__CREATE_DATE
JFPP__IU__UPDATE_DATE          JFPP__IU__CREATE_DATE
JOB__IU__UPDATE_DATE           JOB__IU__CREATE_DATE
ROLE__IU__UPDATE_DATE          ROLE__IU__CREATE_DATE
UR__IU__UPDATE_DATE            UR__IU__CREATE_DATE
OMBT__IU__UPDATE_DATE          OMBT__IU__CREATE_DATE
OMCV__IU__UPDATE_DATE          OMCV__IU__CREATE_DATE
PMPCR__IU__UPDATE_DATE         PMPCR__IU__CREATE_DATE
PMPC__IU__UPDATE_DATE          PMPC__IU__CREATE_DATE
PMP__IU__UPDATE_DATE           PMP__IU__CREATE_DATE
PMPP__IU__UPDATE_DATE          PMPP__IU__CREATE_DATE
PMR__IU__UPDATE_DATE           PMR__IU__CREATE_DATE
PMSMT__IU__UPDATE_DATE         PMSMT__IU__CREATE_DATE
PORSEC__IU__UPDATE_DATE        PORSEC__IU__CREATE_DATE
PPR__IU__UPDATE_DATE           PPR__IU__CREATE_DATE
EMP__IU__UPDATE_DATE           EMP__IU__CREATE_DATE
TRD__IU__UPDATE_DATE           TRD__IU__CREATE_DATE
SEC__IU__UPDATE_DATE           SEC__IU__CREATE_DATE
SYSC__IU__UPDATE_DATE          SYSC__IU__CREATE_DATE
TRL__IU__UPDATE_DATE           TRL__IU__CREATE_DATE
UEVFD__IU__UPDATE_DATE         UEVFD__IU__CREATE_DATE
VOU__IU__UPDATE_DATE           VOU__IU__CREATE_DATE
APRI__IU__UPDATE_DATE          APRI__IU__CREATE_DATE
BP__IU__UPDATE_DATE            BP__IU__CREATE_DATE
BPC__IU__UPDATE_DATE           BPC__IU__CREATE_DATE
BPVEN__IU__UPDATE_DATE         BPVEN__IU__CREATE_DATE
COMP__IU__UPDATE_DATE          COMP__IU__CREATE_DATE
DMIP__IU__UPDATE_DATE          DMIP__IU__CREATE_DATE
DMEIS__IU__UPDATE_DATE         DMIS__IU__CREATE_DATE
EMAL__IU__UPDATE_DATE          EMAL__IU__CREATE_DATE
EMEC__IU__UPDATE_DATE          EMEC__IU__CREATE_DATE
EME__IU__UPDATE_DATE           EME__IU__CREATE_DATE
EMHL__IU__UPDATE_DATE          EMHL__IU__CREATE_DATE
IMGD__IU__UPDATE_DATE          IMGD__IU__CREATE_DATE
OMOP__IU__UPDATE_DATE          OMOP__IU__CREATE_DATE
PMCD__IU__UPDATE_DATE          PMCD__IU__CREATE_DATE
PMCT__IU__UPDATE_DATE          PMCT__IU__CREATE_DATE
PMDIST__IU__UPDATE_DATE        PMDIST__IU__CREATE_DATE
PHY__IU__UPDATE_DATE           PHY__IU__CREATE_DATE
UEV__IU__UPDATE_DATE           UEV__IU__CREATE_DATE
SYSAI__IU__UPDATE_DATE         SYSAI__IU__CREATE_DATE
SYSCR__IU__UPDATE_DATE         SYSCR__IU__CREATE_DATE
UEF__IU__UPDATE_DATE           UEF__IU__CREATE_DATE
UEFP__IU__UPDATE_DATE      UEFP__IU__CREATE_DATE
OMCG__IU__UPDATE_DATE      OMCG__IU__CREATE_DATE
JSGU__IU__CREATE_DATE      JSGU__IU__CREATE_DATE
OMC__IU__CREATE_DATE       OMC__IU__CREATE_DATE
WRL__IU__CREATE_DATE       WRL__IU__CREATE_DATE
INV__IU__CREATE_DATE       INV__IU__CREATE_DATE
JCGJ__IU__CREATE_DATE      JCGJ__IU__CREATE_DATE
OMCT__IU__CREATE_DATE      OMCT__IU__CREATE_DATE
OPUDF__IU__CREATE_DATE     OPUDF__IU__CREATE_DATE
OMOPS__IU__CREATE_DATE     OMOPS__IU__CREATE_DATE
SDPICKR__IU__CREATE_DATE   SDPICKR__IU__CREATE_DATE
VDIST__IU__CREATE_DATE     VDIST__IU__CREATE_DATE
INCIDENT_DATETIME
CORRECTION_DATE
CORRECTION_DATE
UPDATED                    CREATED
UPDATED                    CREATED
CORRECTION_DATE
UPDATED                    CREATED
UPDATED                    CREATED
UPDATED                    CREATED
UPDATED                    CREATED
REFRESH_DATE
UPDATED                    CREATED
UPDATED                    CREATED
UPDATED                    CREATED
UPDATED                    CREATED
TIMESTAMP
UPDATED                    CREATED
UPDATED                    CREATED
UPDATED                    CREATED
UPDATED                    CREATED
CREATED_AT
UPDATED                    CREATED
N/A                        N/A
N/A                        N/A
N/A                        N/A
N/A                        N/A
N/A                        N/A
N/A                        N/A
N/A                        N/A
N/A                        N/A
N/A                        N/A
N/A                        N/A
N/A                        N/A
N/A                        N/A
N/A                        N/A
N/A                               N/A
N/A                               N/A
N/A                               N/A
N/A                               N/A
N/A                               N/A
N/A                               N/A
N/A                               N/A
N/A                               N/A
no watermark column               N/A
created_at,updated_at             N/A
no watermark column               N/A
updated_at                        N/A
updated_at                        N/A
created_at,updated_at             N/A
created_at,updated_at             N/A
created_at,updated_at             N/A
created_at,updated_at             N/A
created_at,updated_at             N/A
no watermark column               N/A
created_at,updated_at             N/A
created_at,updated_at N/A
CREATION_DATE               UPDATE_DATE
CAT__IU__CREATE_DATE        CAT__IU__UPDATE_DATE
FCAT__IU__CREATE_DATE       FCAT__IU__UPDATE_DATE
ORG__IU__CREATE_DATE        ORG__IU__UPDATE_DATE
EMPM__IU__CREATE_DATE       EMPM__IU__UPDATE_DATE
MEMT__IU__CREATE_DATE       MEMT__IU__UPDATE_DATE
SYSCMH__IU__CREATE_DATE     SYSCMH__IU__UPDATE_DATE
JHP__IU__CREATE_DATE        JHP__IU__UPDATE_DATE
OMCT__IU__CREATE_DATE       OMCT__IU__UPDATE_DATE
OMOC__IU__CREATE_DATE       OMOC__IU__UPDATE_DATE
OMC__IU__CREATE_DATE        OMC__IU__UPDATE_DATE
OMOMS__IU__CREATE_DATE      OMOMS__IU__UPDATE_DATE
SDOA__IU__CREATE_DATE       SDOA__IU__UPDATE_DATE
JCBAL__IU__CREATE_DATE      JCBAL__IU__UPDATE_DATE
JCCF__IU__CREATE_DATE       JCCF__IU__UPDATE_DATE
JCCFD__IU__CREATE_DATE      JCCFD__IU__UPDATE_DATE
INSD__IU__CREATE_DATE       INSD__IU__UPDATE_DATE
ESGU_IU_CREATE_DATE         ESGU_IU_UPDATE_DATE
ESG_IU_CREATE_DATE        ESG_IU_UPDATE_DATE
ROLE_IU_CREATE_DATE       ROLE_IU_UPDATE_DATE
SD_IU_CREATE_DATE         SD_IU_UPDATE_DATE
FMR_IU_CREATE_DATE        FMR_IU_UPDATE_DATE
IGU_IU_CREATE_DATE        IGU_IU_UPDATE_DATE
IGR_IU_CREATE_DATE        IGR_IU_UPDATE_DATE
UROLE_IU_CREATE_DATE      UROLE_IU_UPDATE_DATE
SDMC__IU__CREATE_DATE     SDMC__IU__UPDATE_DATE
EMCOM_IU_CREATE_DATE      EMCOM_IU_UPDATE_DATE
UED__IU__CREAT_DATE       UED__IU__UPDATE_DATE
VPDEPT__IU_CREATE_DATE    VPDEPT__IU__UPDATE_DATE
CREATE_TIMESTAMP          LAST_UPDATE_USER
JOB__IU__CREATE_DATE      JOB__IU__UPDATE_DATE
E__IU__CREATE_DATE        E__IU__UPDATE_DATE
PAYTYP__IU__CREATE_DATE   PAYTYP__IU__UPDATE_DATE
Load Type Incremental / Full   Is Full Load Done?   Is WW FL Data Validatation Done?
Yes Yes
Yes Yes
Yes Yes
                               Yes                  Yes
                               Yes                  Yes
Full
Yes Yes
                               Yes                  Yes
                               Yes                  Yes
                               Yes                  Yes
Yes   Yes
Yes   Yes
Yes   Yes
Yes   Yes
Yes   Yes
Yes   Yes
Yes   Yes
Yes   Yes
Yes   Yes
Yes   Yes
Yes   Yes
Yes   Yes
Yes   Yes
Yes   Yes
Yes   Yes
Yes   Yes
Yes   Yes
Yes   Yes
Yes   Yes
Yes   Yes
Yes   Yes
Yes   Yes
Yes   Yes
Yes   Yes
NA    NA
Yes   Yes
Yes   Yes
NA    NA
Yes   Yes
Yes   Yes
Yes   Yes
Yes   Yes
Yes   Yes
Yes   Yes
Yes   Yes
Yes   Yes
       Yes   Yes
       Yes   Yes
       Yes   Yes
       Yes   Yes
       Yes   Yes
       Yes   Yes
       Yes   Yes
       Yes   Yes
       Yes   Yes
       Yes   Yes
       Yes   Yes
       Yes   Yes
       Yes   Yes
       Yes   Yes
       Yes   Yes
       Yes   Yes
       Yes   Yes
       Yes   Yes
       Yes   Yes
       Yes   Yes
Yes Yes
       Yes   Yes
       Yes   Yes
       Yes   Yes
       Yes   Yes
Yes Yes
Yes Yes
       Yes   Yes
       Yes   Yes
Full
Full
Full
Full
Full
Full
Full
Is WT FL Data Validation Done?   Is Incremental Load Done?
Yes Yes
Yes Yes
Yes Yes
Yes                              Yes
Yes                              Yes
Yes Yes
Yes                              Yes
Yes                              Yes
Yes                              Yes
Yes   Yes
Yes   Yes
Yes   Yes
Yes   Yes
Yes   Yes
Yes   Yes
Yes   Yes
Yes   Yes
Yes   Yes
Yes   Yes
Yes   Yes
Yes   Yes
Yes   Yes
Yes   Yes
Yes   Yes
Yes   Yes
Yes   Yes
Yes   Yes
Yes   Yes
Yes   Yes
Yes   Yes
Yes   Yes
Yes   Yes
Yes   N/A
NA    N/A
Yes   N/A
Yes   N/A
NA    N/A
Yes   N/A
Yes   N/A
Yes   N/A
Yes   N/A
Yes   N/A
Yes   N/A
Yes   N/A
Yes   N/A
Yes   N/A
Yes   N/A
Yes   N/A
Yes   N/A
Yes   N/A
Yes   N/A
Yes   N/A
Yes   N/A
YTS   Yes
YTS   Yes
YTS   Yes
YTS   Yes
YTS   Yes
YTS   Yes
YTS   Yes
YTS   Yes
YTS   Yes
YTS   Yes
YTS   Yes
YTS   Yes
YTS Yes
YTS   Yes
YTS   Yes
YTS   Yes
YTS   Yes
YTS Yes
YTS Yes
YTS   Yes
No    Yes
Is WW IL Data Validatation Done?   Is WT IL Data Validation Done?   OPDG - Full Load Sprint #
                                   Yes                                                          10
                                   Yes                                                          10
Yes                                Yes                                                           9
                                   Yes                                                          10
Yes                                Yes                                                           9
                                   Yes                                                          10
                                   Yes                                                          10
                                   Yes                                                          10
                                   Yes                                                          10
                                   Yes                                                          10
Yes                                Yes                                                           9
                                   Yes                                                          10
                                   Yes                                                          10
                                   Yes                                                          10
                                   Yes                                                          10
                                   Yes                                                          10
                                   Yes                                                          10
                                   Yes                                                          10
                                   Yes                                                          10
                                   Yes                                                          10
                                   Yes                                                          10
                                   Yes                                                          10
                                   Yes                                                          10
                                   Yes                                                          10
                                   Yes                                                          10
Yes                                Yes                                                           9
Yes                                Yes                                                           9
                                   Yes                                                          10
                                   Yes                                                          10
                                   Yes                                                          10
                                   Yes                                                          10
                                   Yes                                                          10
                                   Yes                                                          11
                                   Yes                                                          10
                                   Yes                                                          10
                                   Yes                                                          10
                                   Yes                                                          10
Yes                                Yes                                                           9
                                   Yes                                                          10
                                   Yes                                                          10
                                   Yes                                                          10
Yes                                Yes                                                           9
Yes                                Yes                                                           9
                                   Yes                                                          10
Yes                                Yes                                                           9
                                   Yes                                                          10
                                   Yes                                                          10
      Yes   10
      Yes   10
      Yes   10
      Yes   10
      Yes   10
      Yes   10
      Yes   10
      Yes   10
      Yes   10
      Yes   10
      Yes   10
Yes   Yes    9
Yes   Yes
Yes   Yes
Yes   Yes
Yes   Yes
Yes   Yes
Yes   Yes
Yes   Yes
Yes   Yes
Yes   Yes
Yes   Yes
Yes   Yes
Yes   Yes
Yes   Yes
Yes   Yes
Yes   Yes
Yes   Yes
Yes   Yes
Yes   Yes
Yes   Yes
Yes   Yes
Yes   Yes
Yes   Yes
N/A   N/A
N/A   N/A
N/A   N/A
N/A   N/A
N/A   N/A
N/A   N/A
N/A   N/A
N/A   N/A
N/A   N/A
N/A   N/A
N/A   N/A
N/A   N/A
N/A   N/A
N/A   N/A
N/A   N/A
N/A   N/A
N/A   N/A
N/A   N/A
N/A   N/A
N/A   N/A
N/A   N/A
Yes   YTS
Yes   YTS
Yes   YTS
Yes   YTS
Yes   YTS
Yes   YTS
Yes   YTS
Yes   YTS
Yes   YTS
Yes   YTS
Yes   YTS
Yes   YTS
Yes YTS
Yes   YTS
Yes   YTS
Yes   YTS
Yes   YTS
Yes YTS
Yes YTS
Yes   YTS
Yes   No
            11
            10
            10
            10
            10
            10
10
10
10
10
10
10
10
10
10
10
10
10
11
10
10
10
10
11
10
11
10
10
10
10
10
10
10
10
10
10
10
11
10
11
11
11
11
11
11
11
11
11
11
11
11
11
10
10
10
11
11
11
11
11
11
10
11
11
11
11
11
10
11
11
11
11
11
11
11
11
11
11
11
10
10
10
10
10
11
10
11
11
11
11
11
10
11
11
10
11
11
11
11
10
10
10
11
10
10
11
11
11
Is WW completed OPDG - Full Load?   Is WW completed OPDG - Full Load Validation?
Yes                                 Yes
Yes                                 Yes
Yes                                 Yes
Yes                                 Yes
Yes                                 Yes
Yes                                 Yes
Yes                                 Yes
Yes                                 Yes
Yes                                 Yes
Yes                                 Yes
Yes                                 Yes
Yes                                 Yes
Yes                                 Yes
Yes                                 Yes
Yes                                 Yes
Yes                                 Yes
Yes                                 Yes
Yes                                 Yes
Yes                                 Yes
Yes                                 Yes
Yes                                 Yes
Yes                                 Yes
Yes                                 Yes
Yes                                 Yes
Yes                                 Yes
Yes                                 Yes
Yes                                 Yes
Yes                                 Yes
Yes                                 Yes
Yes                                 Yes
Yes                                 Yes
Yes                                 Yes
Yes                                 No
Y                                   Y
Y                                   Y
Y                                   Y
Y                                   Y
Yes                                 Yes
Y                                   Y
Y                                   Y
Y                                   Y
Yes                                 Yes
Yes                                 Yes
Y                                   Y
Yes                                 Yes
Y                                   Y
Y                                   Y
Y     Y
Y     Y
Y     Y
Y     Y
Y     Y
Y     Y
Y     Y
Y     Y
Y     Y
Y     Y
Y     Y
Yes   Yes
Yes   Yes
Y     Y
Y     Y
Y     Y
Y     Y
Y     Y
Y     Y
Y     Y
Y     Y
Y     Y
Y     Y
Y     Y
Y     Y
Y     Y
Y     Y
Y     Y
Y     Y
Y     Y
Yes   Yes
Y     Y
Y     Y
Y     Y
Y     Y
Yes   Yes
Y     Y
Yes   Yes
Y     Y
Y     Y
Y     Y
Y     Y
Y     Y
Y     Y
Y     Y
Y     Y
Y     Y
Y     Y
Y     Y
Yes   Yes
Y     Y
Yes   Yes
Yes   Yes
Yes   Yes
Yes   Yes
Yes   Yes
Yes   Yes
Yes   Yes
Yes   Yes
Yes   Yes
Yes   Yes
Yes   Yes
Yes   Yes
Yes   Yes
Y     Y
Y     Y
Y     Y
Yes   Yes
Yes   Yes
Yes   Yes
Yes   Yes
Yes   Yes
Yes   Yes
Y     Y
Yes   Yes
Yes   Yes
Yes   Yes
Yes   Yes
Yes   Yes
Y     Y
Yes   Yes
Yes   Yes
Yes   Yes
Yes   Yes
Yes   Yes
Yes   Yes
Yes   Yes
Yes   Yes
Yes   Yes
Yes   Yes
Yes   Yes
Y     Y
Y     Y
Y     Y
Y     Y
Y     Y
Yes   Yes
Y     Y
Yes   Yes
Yes   Yes
Yes   Yes
Yes   Yes
Yes   Yes
Y     Y
Yes   Yes
Yes   Yes
Y     Y
Yes   Yes
Yes   Yes
Yes   Yes
Yes   Yes
Y     Y
Y     Y
Y     Y
Yes   Yes
Y     Y
Y     Y
Yes   Yes
Yes   Yes
Yes   Yes
Is WT OPDG FL Data Validation Done?   OPDG - Incremental Load Sprint #
                                                                         10
                                                                         10
Yes                                                                       9
                                                                         10
Yes                                                                       9
                                                                         10
                                                                         10
                                                                         10
                                                                         10
                                                                         10
Yes                                                                       9
                                                                         10
                                                                         10
                                                                         10
                                                                         10
                                                                         10
                                                                         10
                                                                         10
                                                                         10
                                                                         10
                                                                         10
                                                                         10
                                                                         10
                                                                         10
                                                                         10
Yes                                                                       9
Yes                                                                       9
                                                                         10
                                                                         10
                                                                         10
                                                                         10
                                                                         10
                                                                         11
                                                                         10
                                                                         10
                                                                         10
                                                                         10
Yes                                                                       9
                                                                         10
                                                                         10
                                                                         10
Yes                                                                       9
Yes                                                                       9
                                                                         10
Yes                                                                       9
                                                                         10
                                                                         10
      10
      10
      10
      10
      10
      10
      10
      10
      10
      10
      10
Yes    9
11
10
10
10
10
10
10
10
10
10
10
10
10
10
10
10
10
10
11
10
10
10
10
11
10
11
10
10
10
10
10
10
10
10
10
10
10
11
10
11
11
11
11
11
11
11
11
11
11
11
11
11
10
10
10
11
11
11
11
11
11
10
11
11
11
11
11
10
11
11
11
11
11
11
11
11
11
11
11
10
10
10
10
10
11
10
11
11
11
11
11
10
11
11
10
11
11
11
11
10
10
10
11
10
10
11
11
11
Is WW completed OPDG - Incremental Load?
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Y
Y
Y
Y
Yes
Y
Y
Y
Yes
Yes
Y
Yes
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Yes
Yes
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Yes
Y
Y
Y
Y
Yes
Y
Yes
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Yes
Y
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Y
Y
Y
Yes
Yes
Yes
Yes
Yes
Yes
Y
Yes
Yes
Yes
Yes
Yes
Y
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Y
Y
Y
Y
Y
Yes
Y
Yes
Yes
Yes
Yes
Yes
Y
Yes
Yes
Y
Yes
Yes
Yes
Yes
Y
Y
Y
Yes
Y
Y
Yes
Yes
Yes
Is WW completed OPDG - Incremental Load Validation?
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Y
Y
Y
Y
Yes
Y
Y
Y
Yes
Yes
Y
Yes
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Yes
Yes
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Yes
Y
Y
Y
Y
Yes
Y
Yes
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Yes
Y
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Y
Y
Y
Yes
Yes
Yes
Yes
Yes
Yes
Y
Yes
Yes
Yes
Yes
Yes
Y
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Y
Y
Y
Y
Y
Yes
Y
Yes
Yes
Yes
Yes
Yes
Y
Yes
Yes
Y
Yes
Yes
Yes
Yes
Y
Y
Y
Yes
Y
Y
Yes
Yes
Yes
Is WT completed OPDG - Incremental Load Data Validation?
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Notes
Able to parse schema -> prc_admin_equipment_list, good until gold
able to parse schema -> prc_checklist_inspection_schedules_list, good until gold
able to parse schema->prc_checklist_inspections_sections_list, good until gold
able to parse schema-> prc_checklist_sections, good until gold
able to parse schema-> prc_checklist_sections_list, good until gold
in progress. able to parse schema-> prc_construction_report_logs , good until gold
able to parse schema-> prc_forms_list, good until gold
able to parse schema-> prc_incidents, good until gold
able to parse schema-> prc_observations_list, good until gold
able to parse schema-> prc_rfis_list, good until gold
able to parse schema-> prc_meetings_list, in progress
Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: Task 5 in
able
stageto27.0
        parse  schema->
            failed 4 times,prc_project_permission_manifest_list,    in progress
                            most recent failure: Lost task 5.3 in stage  27.0 (TID 189) (vm-
47946987 executor 2): org.apache.spark.SparkException: Parquet column cannot be
converted in file abfss://e165d8f3-7a79-4a3b-b3b8-
fddc1391b133@onelake.dfs.fabric.microsoft.com/59bdf0ab-2f7f-45eb-8124-
d7df81f7628d/Files/bronze_stg/InputFiles/Procore/DOCUMENTS%20LIST/
Procore_Documents%5BList%5D_020406-CyrusOne_041724.parquet. Column:
[created_by.locale], Expected: string, Found: DOUBLE.
ErrorCode=: Parquet complex types of STRUCT, LIST and MAP are not supported.
                                                                                                next
ErrorCode=: Parquet complex types of STRUCT, LIST and MAP are not supported.                    next
Parquet complex types of STRUCT, LIST and MAP are not supported                                 next
Column: [date], Expected: string, Found: DOUBLE
SUBMITTALS%20LIST/Procore_Submittals%5BList%5D_019810-ProjectSCF
%20041924.parquet. Column: [number], Expected: double, Found: BINARY.