Created:
Jul 17, 2024 04:35 AM
| Last Modified: Jul 17, 2024 1:35 AM(456 views)
Hi, I am trying to fetch weather data form XML (Finnish weather service - https://en.ilmatieteenlaitos.fi/open-data-manual-fmi-wfs-services) to JMP in a way that the user could define the time period. My code does not work, I always get the "Invalid HTTP request state" error message.
// User-defined start and end dates (modify as needed)
startDate = "2024-07-10";
endDate = "2024-07-24";
// Construct the URL with user-defined dates and parameters
url = "https://opendata.fmi.fi/wfs?service=WFS&version=2.0.0&request=getFeature&storedquery_id=fmi::observations::weather::timevaluepair&place=helsinki-vantaa_airport×tep=1&starttime=" || startDate || "T00:00:00Z&endtime=" || endDate || "T23:59:59Z¶meters=temperature%2Chumidity%2Cpressure";
// Create HTTP request object
httpRequest = New HTTP Request();
// Send HTTP GET request
response = httpRequest << Send("GET", url);
// Check if the request was successful
If(response << Get Status Code == 200,
// Parse the XML response
xmlResponse = response << Get As XML,
// Extract data from XML response
dataNodes = xmlResponse << XPath("//wml2:MeasurementTVP"),
// Initialize lists to store data
temperatureList = {},
humidityList = {},
pressureList = {},
// Loop through each data node and extract values
ForEach(node in dataNodes,
// Extract timestamp
timestamp = node << XPath("wml2:time"),
// Extract temperature value
temperature = node << XPath("wml2:value[@parameter='temperature']"),
temperatureValue = temperature << NodeText,
// Extract humidity value
humidity = node << XPath("wml2:value[@parameter='humidity']"),
humidityValue = humidity << NodeText,
// Extract pressure value
pressure = node << XPath("wml2:value[@parameter='pressure']"),
pressureValue = pressure << NodeText,
// Convert timestamp to JMP DateTime format
timestampDT = Parse DateTime(timestamp, "YYYY-MM-DDThh:mm:ssZ"),
// Append data to lists
Append(temperatureList, {timestampDT, As Number(temperatureValue)}),
Append(humidityList, {timestampDT, As Number(humidityValue)}),
Append(pressureList, {timestampDT, As Number(pressureValue)})
);
// Create a new JMP data table and add columns
dt = New Table("Helsinki-Vantaa Weather Data",
Add Rows(Num Rows(temperatureList)),
New Column("Timestamp", Numeric, "DateTime"),
New Column("Temperature (°C)", Numeric, "Continuous", Format(3)),
New Column("Relative Humidity (%)", Numeric, "Continuous", Format(3)),
New Column("Pressure (hPa)", Numeric, "Continuous", Format(3))
);
// Populate the data table with values
dt:"Timestamp" << Set Values(temperatureList);
dt:"Temperature (°C)" << Set Values(humidityList);
dt:"Relative Humidity (%)" << Set Values(pressureList);
dt:"Pressure (hPa)" << Set Values(pressureList);
// Show the data table
dt << Show;
,
// Display error if request failed
Throw("Failed to fetch data. HTTP status code: " || Char(response << Get Status Code))
);
Created:
Jul 17, 2024 09:59 AM
| Last Modified: Jul 17, 2024 6:59 AM(375 views)
| Posted in reply to message from jthi 07-17-2024
Below is the script I used (I have used Ilmatieteenlaitos open data in JMP earlier but I think I either used different data or parsed it in a bit different way). It does use some more advanced techniques such as open(char to blob(), "text"), but that can be avoided by just saving the file first as text file. It also has data cleanup and graph creation (verify that the data it creates is correct, I didn't do any checks for that).
Names Default To Here(1);
/*
https://en.ilmatieteenlaitos.fi/open-data
https://en.ilmatieteenlaitos.fi/open-data-manual-accessing-data
*/
location = "helsinki-vantaa_airport";
start_date = "2024-07-10";
end_date = "2024-07-11";
fmi_url = "https://opendata.fmi.fi/wfs?service=WFS&version=2.0.0&request=getFeature&storedquery_id=fmi::observations::weather::timevaluepair&place=^location^×tep=1&starttime=^start_date^T00:00:00Z&endtime=^end_date^T23:59:59Z¶meters=temperature%2Chumidity%2Cpressure";
// Create HTTP request object
request = New HTTP Request(
Method("GET"),
URL(Eval Insert(fmi_url))
);
data = request << send;
dt = Open(
Char to blob(data),
XML Settings(
Stack(0),
Row(
"/wfs:FeatureCollection/wfs:member/omso:PointTimeSeriesObservation/om:result/wml2:MeasurementTimeseries/wml2:point"
),
Col(
"/wfs:FeatureCollection/wfs:member/omso:PointTimeSeriesObservation/om:result/wml2:MeasurementTimeseries/@gml:id",
Column Name(
"MeasurementID"
),
Fill("Use Forever"),
Type("Character"),
Format({"Best"}),
Modeling Type("Continuous")
),
Col(
"/wfs:FeatureCollection/wfs:member/omso:PointTimeSeriesObservation/om:result/wml2:MeasurementTimeseries/wml2:point/wml2:MeasurementTVP/wml2:time",
Column Name(
"Timestamp"
),
Fill("Use Once"),
Type("Character"),
Format({"Best"}),
Modeling Type("Continuous")
),
Col(
"/wfs:FeatureCollection/wfs:member/omso:PointTimeSeriesObservation/om:result/wml2:MeasurementTimeseries/wml2:point/wml2:MeasurementTVP/wml2:value",
Column Name(
"Value"
),
Fill("Use Once"),
Type("Numeric"),
Format({"Best"}),
Modeling Type("Continuous")
)
),
XML Wizard(0),
"text"
);
For Each Row(dt,
:MeasurementID = Word(-1, :MeasurementID, "-");
:Timestamp = Substitute(:Timestamp, "Z", "");
);
Column(dt, "Timestamp") << Data Type(Numeric) << Modeling Type(Continuous) << Format("yyyy-mm-ddThh:mm:ss", 19, 0);
dt_split = dt << Split(
Split By(:MeasurementID),
Split(:Value),
Group(:Timestamp),
Output Table("FMI " || location),
Sort by Column Property
);
Close(dt, no save);
dt_split << Delete Scripts(dt_split << Get Table Script Names);
gb = dt_split << Graph Builder(
Size(857, 524),
Show Control Panel(0),
Variables(X(:Timestamp), Y(:humidity), Y(:pressure), Y(:temperature)),
Elements(Position(1, 1), Line(X, Y, Legend(15))),
Elements(Position(1, 2), Line(X, Y, Legend(16))),
Elements(Position(1, 3), Line(X, Y, Legend(17)))
);
gb << Save Script to Data Table("Plot");
If you wish to see how I filled XML Wizard, you can change the XML Wizard(0), to XML Wizard(1), and run code until that point. You can then see the settings in the GUI I used. And here is some sort of explanation of what those mean
You can get quite far by just using Tall Guess
But it will miss the measurement types and you have to make those few changes I did to repeat them on all rows (you could also fill once -> fill rest in JMP table).
Created:
Jul 17, 2024 08:55 AM
| Last Modified: Jul 17, 2024 5:55 AM(392 views)
| Posted in reply to message from jthi 07-17-2024
Thanks @jthi the HTTP request works now, but it seems that the rest of the script is faulty as well. This is the first time I am trying to get data from XML.
Created:
Jul 17, 2024 09:59 AM
| Last Modified: Jul 17, 2024 6:59 AM(376 views)
| Posted in reply to message from jthi 07-17-2024
Below is the script I used (I have used Ilmatieteenlaitos open data in JMP earlier but I think I either used different data or parsed it in a bit different way). It does use some more advanced techniques such as open(char to blob(), "text"), but that can be avoided by just saving the file first as text file. It also has data cleanup and graph creation (verify that the data it creates is correct, I didn't do any checks for that).
Names Default To Here(1);
/*
https://en.ilmatieteenlaitos.fi/open-data
https://en.ilmatieteenlaitos.fi/open-data-manual-accessing-data
*/
location = "helsinki-vantaa_airport";
start_date = "2024-07-10";
end_date = "2024-07-11";
fmi_url = "https://opendata.fmi.fi/wfs?service=WFS&version=2.0.0&request=getFeature&storedquery_id=fmi::observations::weather::timevaluepair&place=^location^×tep=1&starttime=^start_date^T00:00:00Z&endtime=^end_date^T23:59:59Z¶meters=temperature%2Chumidity%2Cpressure";
// Create HTTP request object
request = New HTTP Request(
Method("GET"),
URL(Eval Insert(fmi_url))
);
data = request << send;
dt = Open(
Char to blob(data),
XML Settings(
Stack(0),
Row(
"/wfs:FeatureCollection/wfs:member/omso:PointTimeSeriesObservation/om:result/wml2:MeasurementTimeseries/wml2:point"
),
Col(
"/wfs:FeatureCollection/wfs:member/omso:PointTimeSeriesObservation/om:result/wml2:MeasurementTimeseries/@gml:id",
Column Name(
"MeasurementID"
),
Fill("Use Forever"),
Type("Character"),
Format({"Best"}),
Modeling Type("Continuous")
),
Col(
"/wfs:FeatureCollection/wfs:member/omso:PointTimeSeriesObservation/om:result/wml2:MeasurementTimeseries/wml2:point/wml2:MeasurementTVP/wml2:time",
Column Name(
"Timestamp"
),
Fill("Use Once"),
Type("Character"),
Format({"Best"}),
Modeling Type("Continuous")
),
Col(
"/wfs:FeatureCollection/wfs:member/omso:PointTimeSeriesObservation/om:result/wml2:MeasurementTimeseries/wml2:point/wml2:MeasurementTVP/wml2:value",
Column Name(
"Value"
),
Fill("Use Once"),
Type("Numeric"),
Format({"Best"}),
Modeling Type("Continuous")
)
),
XML Wizard(0),
"text"
);
For Each Row(dt,
:MeasurementID = Word(-1, :MeasurementID, "-");
:Timestamp = Substitute(:Timestamp, "Z", "");
);
Column(dt, "Timestamp") << Data Type(Numeric) << Modeling Type(Continuous) << Format("yyyy-mm-ddThh:mm:ss", 19, 0);
dt_split = dt << Split(
Split By(:MeasurementID),
Split(:Value),
Group(:Timestamp),
Output Table("FMI " || location),
Sort by Column Property
);
Close(dt, no save);
dt_split << Delete Scripts(dt_split << Get Table Script Names);
gb = dt_split << Graph Builder(
Size(857, 524),
Show Control Panel(0),
Variables(X(:Timestamp), Y(:humidity), Y(:pressure), Y(:temperature)),
Elements(Position(1, 1), Line(X, Y, Legend(15))),
Elements(Position(1, 2), Line(X, Y, Legend(16))),
Elements(Position(1, 3), Line(X, Y, Legend(17)))
);
gb << Save Script to Data Table("Plot");
If you wish to see how I filled XML Wizard, you can change the XML Wizard(0), to XML Wizard(1), and run code until that point. You can then see the settings in the GUI I used. And here is some sort of explanation of what those mean
You can get quite far by just using Tall Guess
But it will miss the measurement types and you have to make those few changes I did to repeat them on all rows (you could also fill once -> fill rest in JMP table).
Created:
Jul 18, 2024 04:27 AM
| Last Modified: Jul 18, 2024 1:27 AM(312 views)
| Posted in reply to message from jthi 07-17-2024
I tried the import XML add-in in JMP before but I got lost with it, however, now based on your explanation I understand how it works! @jthi Thank you! Also, I really appreciate that you shared the way you fetched this data using open(char to blob(), "text")!