Data Transformer Task - standard functions support

[phertzog]

Hello,

I have written a jsonnet mapping to add properties to json object. The purpose is to add new properties to json object if they are missing because the input csv didn't contain the expected column. The code is following

local f = import "functions"; // Import additional functions
local csv_content = f.extVar("file_content");
local mapping = f.extVar("csv_mapping");
local json_content = f.parseCsvWithHeader(csv_content, ";");
local translate(item, map) =
  item { [q.value]: if std.objectHas(item, q.key) then item[q.key] else '??' for q in std.objectKeysValues(map) };

// TEMPLATE OUTPUT
{
   orders: [translate(item, mapping) for item in json_content],
}

I also checked by replacing std. with f. (alias of the functions import) but it didn't work neither.

  When I run the integration I get following error : 

RUNTIME ERROR: Field does not exist: objectKeysValues template:6:87-107 function template:6:8-114 template:6:8-114 template:15:13-37 thunk from > Array element 0 Field "orders" During manifestation : invalid argument: invalid argument

Based on this document (https://jsonnet.org/ref/stdlib.html) objectKeysValues is a std function. The Integration documentation specifies that the std library is supported

To use the following predefined functions, you must import the functions library in your script. Importing the functions library lets you use both the standard Jsonnet functions and the predefined Data Transformer functions.

For information about the Jsonnet standard functions, see Jsonnet Standard Library.

[andyhood · Accepted solution]

Hi @phertzog,
Whilst the vast majority of the Std library functions are implemented, unfortunately it seems that std.objectKeysValues() and std.objectKeysValuesAll() haven't yet been. This has been raised with our Engineering team and hopefully an update will be forthcoming.

Regards,
Andy

[robertcarlos]

Hi @phertzog,

Welcome to Google Cloud Community!

As of this writing, Data Transformer Script task is still in Pre-GA Offerings Terms (preview), meaning that support for this product/service is still limited and may encounter errors along the way.

As of the moment, you may refer to the following documentations for reference until this service would be generally available in the near future:

• Data mapping overview 
• Data Transformer functions 
• Data Transformer Script task 
• Intuitive data mapping using Data Transformer (Script) task 

You may also file a bug so that our engineers could take a look at this. We don't have a specific ETA for this but you can keep track of its progress once the ticket has been created.

Hope this helps.

[andyhood · Accepted solution]

Hi @phertzog,
Whilst the vast majority of the Std library functions are implemented, unfortunately it seems that std.objectKeysValues() and std.objectKeysValuesAll() haven't yet been. This has been raised with our Engineering team and hopefully an update will be forthcoming.

Regards,
Andy

[phertzog]

Hello,

Thanks for you feedback. I found another way around this missing feature.

Regards.

Philippe

[andyhood · Accepted solution]

Hi Philippe,

FYI, our engineering team have just rolled out an update and the objectKeysValues() and objectKeysValuesAll() standard library functions are now available. They are also available using the functions alias , e.g f.objectKeysValues()

Regards,

Andy

[pramodvallanur]

That was fast from a request to delivery within a week. Thanks for the update Andy

[phertzog]

Thanks, and as other said, impressive reaction time.

Philippe

[LalamSiri]

@andyhood I am trying to use data transformer script to convert xml to json and json to csv. Can you provide me a sample script that takes xml from REST API response.

[phertzog]

Hello,

You have a function that parses Xml to Json using a format explained in the following document : http://www.sklar.com/badgerfish/

Based on the example above, you would use parseXml function

local f = import "functions"; // Import additional functions
local xml_content = f.extVar("xml_content");
local json_content = f.parseXml(xml_content, format = "badgerfish");

Didn't test it but I think this should work for going from XML to JSON