[lacnog] LACNIC RPKI RRDP unexpected delta mutation

[Job Snijders]

Dear all,

Something very strange happened (times in UTC):

Oct 17 23:23:15 rpki-client: https://rrdp.lacnic.net/rrdp/notification.xml: a5ea60b9-fd0d-4664-999a-7fcc801a6ae1#101 unexpected delta mutation (expected 7F894B30AEEC0048D2EE2311789737E57143FB16DF1BCECEA56ACA55BA9FEC0A, got EE89EE6581F48C358DE34EA04FED197778C333F09463BED53C670BCF4632E0CB)

Below is a time series overview of what https://rrdp.lacnic.net/rrdp/notification.xml
indicated the hash for delta 101 should be:

2023-10-17T22:24:02  <delta serial="101" uri="https://rrdp.lacnic.net/rrdp/a5ea60b9-fd0d-4664-999a-7fcc801a6ae1/101/delta.xml" hash="ee89ee6581f48c358de34ea04fed197778c333f09463bed53c670bcf4632e0cb"/>
2023-10-17T22:31:02  <delta serial="101" uri="https://rrdp.lacnic.net/rrdp/a5ea60b9-fd0d-4664-999a-7fcc801a6ae1/101/delta.xml" hash="ee89ee6581f48c358de34ea04fed197778c333f09463bed53c670bcf4632e0cb"/>
2023-10-17T22:38:03  <delta serial="101" uri="https://rrdp.lacnic.net/rrdp/a5ea60b9-fd0d-4664-999a-7fcc801a6ae1/101/delta.xml" hash="ee89ee6581f48c358de34ea04fed197778c333f09463bed53c670bcf4632e0cb"/>
2023-10-17T22:52:01  <delta serial="101" uri="https://rrdp.lacnic.net/rrdp/a5ea60b9-fd0d-4664-999a-7fcc801a6ae1/101/delta.xml" hash="ee89ee6581f48c358de34ea04fed197778c333f09463bed53c670bcf4632e0cb"/>
2023-10-17T22:59:02  <delta serial="101" uri="https://rrdp.lacnic.net/rrdp/a5ea60b9-fd0d-4664-999a-7fcc801a6ae1/101/delta.xml" hash="ee89ee6581f48c358de34ea04fed197778c333f09463bed53c670bcf4632e0cb"/>
2023-10-17T23:06:02  <delta serial="101" uri="https://rrdp.lacnic.net/rrdp/a5ea60b9-fd0d-4664-999a-7fcc801a6ae1/101/delta.xml" hash="7f894b30aeec0048d2ee2311789737e57143fb16df1bcecea56aca55ba9fec0a"/>
2023-10-17T23:27:02  <delta serial="101" uri="https://rrdp.lacnic.net/rrdp/a5ea60b9-fd0d-4664-999a-7fcc801a6ae1/101/delta.xml" hash="7f894b30aeec0048d2ee2311789737e57143fb16df1bcecea56aca55ba9fec0a"/>
2023-10-17T23:41:03  <delta serial="101" uri="https://rrdp.lacnic.net/rrdp/a5ea60b9-fd0d-4664-999a-7fcc801a6ae1/101/delta.xml" hash="7f894b30aeec0048d2ee2311789737e57143fb16df1bcecea56aca55ba9fec0a"/>
2023-10-17T23:47:02  <delta serial="101" uri="https://rrdp.lacnic.net/rrdp/a5ea60b9-fd0d-4664-999a-7fcc801a6ae1/101/delta.xml" hash="7f894b30aeec0048d2ee2311789737e57143fb16df1bcecea56aca55ba9fec0a"/>
2023-10-17T23:54:02  <delta serial="101" uri="https://rrdp.lacnic.net/rrdp/a5ea60b9-fd0d-4664-999a-7fcc801a6ae1/101/delta.xml" hash="7f894b30aeec0048d2ee2311789737e57143fb16df1bcecea56aca55ba9fec0a"/>

As can be seen, at 2023-10-17T23:06:02 the SHA256 hash 'retroactively'
is changed to a different value. RRDP deltas are expected to be
immutable after production, and the above data suggests that something
very strange happened because delta 101 mutated. I observed this on
multiple validators in different locations.

The RRDP session possibly needs to be reset to help clients that are
unable to detect this type of RRDP session corruption and now are
possibly in a desynchronized state.

Can LACNIC investigate what transpired?

Kind regards,

Job