Empty plot without error when swapping axis in compare + interpolate mode

[olivhoenen]

This is a follow-up of #76, using the state compare-issueIbexState.json.

Main use case now works (very slow but this is a different topic):

It just seems to have a problem when swapping axis (time in x instead of rho_tor_norm): this shows the first rho_tor_norm=0 ok

but other radial points show empty plot. Also note that starting from a single URI, with time as x-axis, it complains with "coordinates do not match or are missing" warning, while I add other one or two URIs with the same electrons/temperature field, while doing the same operations when the x-axis is rho_tor_norm works fine.

Originally posted by @olivhoenen in #99 (comment)

[mfiaudrin-epsyl]

Hello @olivhoenen,

I have checked the data received and the interpolation always return NaN elements in the yData matrix (whether we transpose x/y axis or not). I suppose this issue is more relevant to BE.

@jwasikpsnc, could you please check if anything goes wrong with Olivier's dataset?

However, I have done more tests to ensure the transposition works as expected with the interpolation and I have found a bug: when we transpose axis and then interpolate, the axis transposition is preserved but the data aren't transposed. I create an issue and mark it as ready once it'll be fixed.

[bpalak]

Simplifying the case:

export FIRST=imas:hdf5?path=/work/imas/shared/imasdb/ITER/3/120120/1#core_profiles:0/profiles_1d[:]/electrons/temperature
export SECOND=imas:hdf5?path=/work/imas/shared/imasdb/ITER/3/100020/1#core_profiles:0/profiles_1d[:]/electrons/temperature
export THIRD=imas:hdf5?path=/work/imas/shared/imasdb/ITER/3/100120/1#core_profiles:0/profiles_1d[:]/electrons/temperature
curl -sS -G http://127.0.0.1:8123/data/plot_data     --data-urlencode "uri=$FIRST"     --data-urlencode "interpolate_over=$SECOND"     --data-urlencode "interpolation_method=exact_value"    --data-urlencode "interpolate_over=$THIRD"     --data-urlencode "interpolation_method=exact_value"        | jq

[bpalak]

TL;DR: It seems that presented plots/heatmaps are... perfectly fine. The plots for rho_tor_norm != 0 are NOT EMPTY. Most (every?) of them displays few (1 or 2 or...) pixels :).

Analysis:
Initial data:

• Three signals - every of shape (175, 21)
• Every signal coordinates:
  • rho_tor_norm - (175, 21)
  • time (175)
• Values of time overlaps, while rho_tor_norm tuples are mostly distinct/unique

Two subsequent 'resamplings' lead to the very LARGE and very SPARSE (!) matrix:

• to produce a heatmap, the frontend generally needs a rectangular matrix: Z[time_index][rho_index]
• small floating-point differences make thousands of radial coordinates distinct
• shape (175, 9970): 175 time coordinates x 9,977 distinct rho_tor_norm coordinates = 1,745,975 cells per signal (BE response of size about 80MB, 6mlns of lines)
• rho=0 is special because zero is represented identically across the profiles.
• Other radial positions usually vary slightly with time, creating thousands of nearly identical but distinct columns. E.g. rho=0.0384... produces only one populated pixel.
• Most nonzero columns similarly contain very few populated pixels.
  • There are 3,500 known nonzero-rho temperature samples.
  • They occupy 3,326 distinct radial columns.
  • Therefore, most radial coordinates appear at only one time.

[olivhoenen]

I get that rho=0 and rho=1 are specific values that are present in all 3 URIs, so plot is showing the 3 curves there
I don't understand why for other rho values we don't see a single curve, as this rho shall come from an existing value in at least one of the URIs.... (it's using by default the exact value isn't it?)

If I look at the first non 0 rho value (in the plot where I have all three curves), it gives 0.0459936745186146

while first non 0 value in URI-0 is 0.04921817330181093, in URI-1 it is 0.049283987917519714 and in URI-2 it is 0.049298240817181246

[jwasikpsnc]

Hi @olivhoenen
After long investigation, i came to conclusion that the problem lies in frontend and it's not what you except.

First of all, how IBEX BE returns data through plot_data endpoint
It returns data array (num dimensions according to coordinates count) and coordinates.
Coordinates can be dependend on others. In our case rho_tor_norm is connected to time, and we list time as rho_tor_norm coordinate (cooridinate of the coordinate).
If interpolate_over is empty, we return full array of given coordinate. If rho_tor_norm is 1D, returned array will be 2D, where one of the dimension is time.

...
name:temperature
value:[[1,2,3],[4,5,6][7,8,9]] (simple 3x3 array) 
coordinates:[
{
  name:rho_tor_norm,
  value:[
    [0, 0.1, 0.2],          # <- rho_tor_norm value for time=1
    [0, 0.11, 0.22],     # <- rho_tor_norm value for time=2
    [0, 0.111, 0.222] # <- rho_tor_norm value for time=3
    # ^ please note that rho_tor_norm value array is 2D
  ]
},
{
  name:time,
  value:[
    [1,2,3]
  ]
}
]

In this case, if X array contains rho_tor_norm and time-slider is set to 1, you will have plotted line through points (0, 1) (0.1, 2) (0.2, 3) <- (X,Y) i.e. (rho_tor_norm_value, temperature_value)

BUT

if set x_axis = time
X axis contains values [1,2,3], and if slider (rho_tor_norm) is set to 0, line will go through 3 points (because there is rho_tor_norm=0 for all time slices) -> (1, 0) (2, 0) (3, 0) <- (X,Y) i.e. (time_value, temperature_value)
^^ this is the case we get when rho_tor_norm slider is set to 0 in the compare-issueIbexState.json.

AND HERE THOUBLE STARTS

if set x_axis = time
X axis contains values [1,2,3], and if slider (rho_tor_norm) is set to 0.1, line will have only one point (because there is rho_tor_norm=0.1 only for one time slice)

You can check how many unique values URI-0 rho_tor_norm has by executing this script:

import requests

url = "http://localhost:40443/data/plot_data"

FIRST="imas:hdf5?path=/work/imas/shared/imasdb/ITER/3/120120/1#core_profiles:0/profiles_1d[:]/electrons/temperature"
params = {
    "uri":FIRST,
    "interpolation_method":"exact_value",
}
response = requests.get(url, params=params)

if response.status_code != 200:
    raise Exception(f"ERROR: {response.status_code}")

data = response.json()["data"]
rho_tor_norm_value = data["coordinates"][0]["value"]

num_rho_tor_norm_elements = sum(len(row) for row in rho_tor_norm_value)
print(f"=== RHO_TOR_NORM HAS {num_rho_tor_norm_elements} ELEMENTS")

unique_rho_tor_norm_elements = set(x for row in rho_tor_norm_value for x in row)
print(f"=== RHO_TOR_NORM HAS {len(unique_rho_tor_norm_elements)} UNIQUE ELEMENTS")

(venv) ~/Desktop/work/IBEX$ python check_issue_103.py 
=== RHO_TOR_NORM HAS 3675 ELEMENTS
=== RHO_TOR_NORM HAS 3327 UNIQUE ELEMENTS

^^ this is URI-0 rho_tor_norm without interpolation.
After interpolation you will get even more sparse array.

In my opinion there is bug in frontend and it should not also display plot at all when switching x-axis to time. (because of not enough points co connect)

The way BE returns coordinates as presented is implemented from very beginning of the project. Probably it wasn't found, because most of the coordinates values set don't change when changing other coordinate (more explanation in #99).

[bpalak]

@jwasikpsnc is right concerning the 'sparse' nature of an array as the reason of the problem. However for a fixed time there are some valid points among 9977 entries to be displayed.
All three URIs use the same 175 exact time coordinates but as Jedrzej pointed out - there are 3327 unique rho values. They differ just a bit, but ... differ. So if you select the fixed rho value, the response will contain just 1 temp value - 3325 rho columns (out of 3327) have exactly one populated time. And 1 value makes plotting a curve impossible.