2024.3.13

[June17 Upd] Re-release of and other issues found in “5-km resolution ensemble climate data in Japan.”

[Updated on June 17]

Regarding the data issue reported below, we have checked all the experiments and years other than those replaced by May 17 and found that two new experiments and years had the same problems.

We have recalculated and have replaced all of them by today. We have retained the previous data as “.YYYYY” with a period at the beginning of the directory, but please refrain from using them.

Experiments and years that have been replaced due to confirmed problems:

(Replaced on June 17)

  •     HPB_m004: 1987
  •     HFB_2K_MR m102 : 2059

(Replaced on May 17)

  •     HFB_4K_MP_m101 : 2079 , 2080, 2082, 2084, 2085, 2103
  •     HFB_4K_CC_m102 : 2051
  •     HFB_2K_GF_m102: 2069
  •     HFB_2K_HA_m101: 2039
  •     HFB_2K_MP_m102: 2087
  •     HPB_m004: 1978, 1982, 1983, 1984, 1986

We apologize for any inconvenience caused by the issue.

If you have any other concerns about the data, please contact us.


[Updated on May 17]

Regarding the data problems reported below, we have recalculated these and replaced them with the correct data. The previous data are still stored with a period at the beginning of the directory as “.YYYY,” but please refrain from using them.

The cause of this issue seems to be that when the computer being used stopped and restarted for its maintenance, etc., the calculations continued using incorrect information. This is the reason why there are many errors in the near year of a particular member.

In addition to the above, we found that the data below has the same issue. This will be replaced in the future, yet the timing has not yet been determined. As above, the period is added at the beginning of the directory name, so please refrain from using this data.

Experiments and years in which defects were newly identified

  •     HFB_2K_MR m102 : 2059

We deeply understand the inconvenience these data issues may have caused and sincerely ask for your patience as we work to resolve them.


[Updated on Apr. 26]

We would like to inform you of an issue with some of the data below.

We had announced that the replacement of these data was expected in mid-April, but due to the extra time required for recalculation, the replacement is now expected to be completed in early to mid-May.

We apologize for any inconvenience and ask for your understanding and patience.


[Reported on Mar.13]

As announced on February 18, 2024, the “5-km resolution ensemble climate data in Japan” (d4PDF_5kmDDS_JP) has been temporarily unavailable. However, we found no defects in the netCDF file date information after investigation, so we are now resuming its release.

Please refer to “1. netCDF date and calendar issue” below for the handling of date information.

In addition to the date issue above, we have found another problem in several data. We plan to replace it in the future. Until then, please refrain from using these data. For details, please refer to “2. Other issue in some data.”

1. netCDF date and calendar issue

A Python user asked us if the initial date in netCDF was incorrect. The data provider confirmed that the date was off by two days when read in Python or Matlab. Therefore, we temporarily stopped releasing the data, and the data provider and DIAS personnel investigated the netCDF data on DIAS.

First, we found that this symptom was only present in the 1-hour 2-dimensional variables (hourly surf and ph2m) and the 30-minute precipitation (30min).

The netCDF date information for these data is recorded as “standard” calendar type and the date as the total time since January 1, 1 AD.” The “standard” calendar is a mixture of the Gregorian and Julian calendars, switching from Julian to Gregorian in October 1587 (see Gregorian and Julian calendars for more information). We have confirmed that the recorded dates are correct under this calendar.

The data provider and DIAS personnel have confirmed that GrADS, NCAR NCL, Python+Xarray, etc., can read netCDF’s standard calendar correctly. In contrast, the Python “datetime” library and Matlab adopted the Proleptic Gregorian calendar, which applied the Gregorian calendar even before 1587, and we found that using these libraries resulted in a 2-day discrepancy.

Based on the above investigation, we have determined that there is no issue with the data on DIAS itself, and we are re-releasing the data in its original form without replacing it.

Please refer to the FAQ for possible measures for Python users. Since we have not found a Matlab library that can handle the data correctly, please create a program to calculate the date by yourself or use the data as if the initial time is July 24, 00Z.

2. Other issue in some data

We found an issue in the data in the following years and experiments. There are some negative values for precipitation.

Experiments and years in which issues were found are:

  • HFB_4K_MP_m101: 2079, 2080, 2082, 2084, 2085, 2103
  • HFB_4K_CC_m102: 2051
  • HFB_2K_GF_m102: 2069
  • HFB_2K_HA_m101: 2039
  • HFB_2K_MP_m102: 2087
  • HPB_m004: 1978, 1982, 1983, 1984, 1986

For these experiments and years, the directories are prefixed with “.” and renamed “.YYYY” (YYYY is the year). Please do not use these data.

The data provider is currently undergoing recalculations, and these data will be replaced upon completion. We expect to replace these data in mid-April.

We apologize for any inconvenience.