Issue running healthcare data harmonization pipeline

[eSully]

Hi - Was wondering if anyone could offer some help/guidance on an issue I'm hitting. New to GCP and Dataflow. I'm trying to run a DataFlow job for HL7v2 to FHIR pipeline from healthcare data harmonization repo (https://github.com/GoogleCloudPlatform/healthcare-data-harmonization-dataflow#how-to-run).

The required infra is setup, the Jar for the pipeline builds fine now and also running the pipeline as above (with subnet added) initially runs fine and shows a running "streaming" job on the dataflow jobs page. However it never reads or processes data from my Pub/Sub subscription. when I interrogate Logs Explorer I see the error below (which doesn't show on the job page).

Any help/pointers appreciated.

Regards 

{
  "insertId": "s=abfde3644eb5496f900e02c8eb92d71f;i=1b40;b=7cc8066fe1f14c9db038e06c130dccbb;m=6d30b1bc4;t=5eeb2c42b8840;x=9d0faff6d91c33e6",
  "jsonPayload": {
    "message": "\"Error syncing pod, skipping\" err=\"failed to \\\"StartContainer\\\" for \\\"java-streaming\\\" with ImagePullBackOff: \\\"Back-off pulling image \\\\\\\"gcr.io/cloud-dataflow/v1beta3/beam-java17-streaming:2.35.0\\\\\\\"\\\"\" pod=\"default/df-hl7v2tofhirstreamingrunne-11300031-q89m-harness-53dh\" podUID=6016f04e50d7b3cf9be7c8c30b439480",
    "line": "pod_workers.go:918",
    "thread": "775"
  },
  "resource": {
    "type": "dataflow_step",
    "labels": {
      "project_id": "ndr-discovery",
      "region": "europe-west2",
      "job_id": "2022-11-30_00_31_32-12431633722710717950",
      "job_name": "hl7v2tofhirstreamingrunner-gigdhc0cl0eu125435-1130083122-2b0132c9",
      "step_id": ""
    }
  },
  "timestamp": "2022-11-30T16:40:46.329880Z",
  "severity": "ERROR",
  "labels": {
    "compute.googleapis.com/resource_name": "hl7v2tofhirstreamingrunne-11300031-q89m-harness-53dh",
    "dataflow.googleapis.com/log_type": "system",
    "dataflow.googleapis.com/region": "europe-west2",
    "compute.googleapis.com/resource_type": "instance",
    "compute.googleapis.com/resource_id": "7444062325900487659",
    "dataflow.googleapis.com/job_id": "2022-11-30_00_31_32-12431633722710717950",
    "dataflow.googleapis.com/job_name": "hl7v2tofhirstreamingrunner-gigdhc0cl0eu125435-1130083122-2b0132c9"
  },
  "logName": "projects/ndr-discovery/logs/dataflow.googleapis.com%2Fkubelet",
  "receiveTimestamp": "2022-11-30T16:40:50.934513594Z"
}

[anjelab]

This seems to be not reproducible on my end. If you have premium support, you can check with GCP Support to further check your issue since this is specific to your project.

[eSully]

Hi Anjelab

Thanks for getting back to me. We are in the process of procuring for our org but don't have premium support yet. I'm confused though, can you confirm what steps/versions you ran the pipeline with. I'm running it exactly as it recommends in the repo and readme. There's no customisations or code changes but I'm getting issues pulling the beam flow image by the looks, did you use a newer release of Java when doing the build? I'm getting "not found: manifest unknown: Failed to fetch "2.35.0" from request "/v2/cloud-dataflow/v1beta3/beam-java17-streaming/manifests/2.35.0"

jsonPayload: {
line: "container_images.go:106"
message: "Checking docker image integrity of gcr.io/cloud-dataflow/v1beta3/beam-java17-streaming:2.35.0"
}

jsonPayload: {
line: "checks.go:162"
message: "Container image gcr.io/cloud-dataflow/v1beta3/beam-java17-streaming:2.35.0 not downloaded yet."
}

jsonPayload: {
line: "kuberuntime_manager.go:918"
message: "container &Container{Name:java-streaming,Image:gcr.io/cloud-dataflow/v1beta3/beam-java17-streaming:2.35.0,Command:[],Args:[--log_file=/var/log/dataflow/boot-json.log --log_dir=/var/log/dataflow --work_dir=/var/opt/google/dataflow --tmp_dir=/var/opt/google/tmp --endpoint=https://dataflow.googleapis.com/ --physmem_limit_pct=37],WorkingDir:,Ports:[]ContainerPort{ContainerPort{Name:j-jmx-port,HostPort:5555,ContainerPort:5555,Protocol:TCP,HostIP:,},ContainerPort{Name:j-status-port,HostPort:8081,ContainerPort:8081,Protocol:TCP,HostIP:,},},Env:[]EnvVar{},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{memory: {{6529482752 0} {<nil>} 6529482752 DecimalSI},},},VolumeMounts:[]VolumeMount{VolumeMount{Name:java-streaming-logs,ReadOnly:false,MountPath:/var/log/dataflow,SubPath:,MountPropagation:nil,SubPathExpr:,},VolumeMount{Name:persist,ReadOnly:false,MountPath:/var/opt/google,SubPath:,MountPropagation:nil,SubPathExpr:,},VolumeMount{Name:metadata,ReadOnly:true,MountPath:/var/lib/dataflow,SubPath:,MountPropagation:nil,SubPathExpr:,},VolumeMount{Name:nvidia-drivers-volume,ReadOnly:false,MountPath:/usr/local/nvidia,SubPath:,MountPropagation:nil,SubPathExpr:,},},LivenessProbe:nil,ReadinessProbe:nil,Lifecycle:nil,TerminationMessagePath:/dev/termination-log,ImagePullPolicy:IfNotPresent,SecurityContext:nil,Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,} start failed in pod df-hl7v2tofhirstreamingrunne-12020110-9ttf-harness-1j96_default(6016f04e50d7b3cf9be7c8c30b439480): ErrImagePull: rpc error: code = Unknown desc = Error response from daemon: manifest for gcr.io/cloud-dataflow/v1beta3/beam-java17-streaming:2.35.0 not found: manifest unknown: Failed to fetch "2.35.0" from request "/v2/cloud-dataflow/v1beta3/beam-java17-streaming/manifests/2.35.0"."
thread: "767"
}

jsonPayload: {
line: "pod_workers.go:918"
message: ""Error syncing pod, skipping" err="failed to \"StartContainer\" for \"java-streaming\" with ErrImagePull: \"rpc error: code = Unknown desc = Error response from daemon: manifest for gcr.io/cloud-dataflow/v1beta3/beam-java17-streaming:2.35.0 not found: manifest unknown: Failed to fetch \\\"2.35.0\\\" from request \\\"/v2/cloud-dataflow/v1beta3/beam-java17-streaming/manifests/2.35.0\\\".\"" pod="default/df-hl7v2tofhirstreamingrunne-12020110-9ttf-harness-1j96" podUID=6016f04e50d7b3cf9be7c8c30b439480"
thread: "767"
}

[Rajappar]

Hi eSully,

Did you find any fix to this issue, as we are facing something very similar.Thanks in advance for your help.

thanks,

Raj

[eSully · Accepted solution]

Sorry Raj, only just seen this. It was a JDK issue if I recall, I was building with too new a version of the jdk and that was causing an issue when starting the 2.35.0 dataflow image. I had a separate response as below:

I understand that you have an issue where the pipeline looks like it's running but not writing to storage. Let me know if I misunderstood.

After investigating your issue I have found that the error you are getting is due to using the Java Development Kit on version 17. A possible solution is to change the Java version. 

To change the Java version do: “sudo update-alternatives --config java”. Then select the JDK 11 version by introducing the corresponding number (in my case I entered ‘1’) and then press enter.
To change the Java compiler version do: “sudo update-alternatives --config javac”. Again, select the JDK 11 version by introducing the corresponding number (in my case I entered ‘1’ as in the previous case) and then press enter.