[lacnog] nightly LACNIC RPKI repository problems?

[Job Snijders]

Dear all,

It appears twice a day there has been some kind of issue which renders
the whole LACNIC RPKI repository unusable for RPKI caches.

This means that twice a day all LACNIC networks flip to "RPKI not-found"
despite having created RPKI ROAs to make their announcements 'valid'.

The 'fgrep' command below shows that around 15:04 UTC the LACNIC VRP
count dropped to 0, and came back the next run at 15:24, then it was
broken again at 23:03, and came back in the subsequent run 15 minutes
later.

Can anyone shed light on what is happening?

Kind regards,

Job

job en tangier:/tank/rpkirepositories/.zfs/snapshot$ fgrep -c lacnic 202005*/output/csv | grep -1 :0 | tail -23
20200517-1445/output/csv:8787
20200517-1504/output/csv:0
20200517-1524/output/csv:8779
--
20200517-2243/output/csv:8788
20200517-2303/output/csv:0
20200517-2323/output/csv:8780
--
20200518-0643/output/csv:8793
20200518-0703/output/csv:0
20200518-0723/output/csv:8793
--
20200518-1443/output/csv:8793
20200518-1503/output/csv:0
20200518-1523/output/csv:8795
--
20200518-2243/output/csv:8808
20200518-2303/output/csv:0
20200518-2323/output/csv:8808
--
20200519-0643/output/csv:8810
20200519-0703/output/csv:0
20200519-0723/output/csv:8494

more detailed log output:

May 18 23:01:39 run-validation: rpki-client: repository.lacnic.net/rpki: loading
May 18 23:01:45 run-validation: rpki-client: repository.lacnic.net/rpki: loaded
May 18 23:01:46 run-validation: rpki-client: repository.lacnic.net/rpki/lacnic/48f083bb-f603-4893-9990-0284c04ceb85/ff14e9055d5afaa37fbe20f4a26bd13c8f18d79a.mft: referenced file 31ef1c6ecf5c293c01f2c63f43f6ba98ebf024f8.cer: No such file or directory
May 18 23:03:19 run-validation: Found valid trust anchor rsync://repository.lacnic.net/rpki/lacnic/rta-lacnic-rpki.cer. Processing.
May 18 23:03:19 run-validation: rsync://repository.lacnic.net/rpki/lacnic/48f083bb-f603-4893-9990-0284c04ceb85/31ef1c6ecf5c293c01f2c63f43f6ba98ebf024f8.cer: not found in local repository
May 18 23:03:19 run-validation: rsync://repository.lacnic.net/rpki/lacnic/48f083bb-f603-4893-9990-0284c04ceb85/31ef1c6ecf5c293c01f2c63f43f6ba98ebf024f8.cer: failed to load.
May 18 23:03:33 run-validation: Found valid trust anchor rsync://repository.lacnic.net/rpki/lacnic/rta-lacnic-rpki.cer. Processing.
May 18 23:03:33 run-validation: rsync://repository.lacnic.net/rpki/lacnic/48f083bb-f603-4893-9990-0284c04ceb85/31ef1c6ecf5c293c01f2c63f43f6ba98ebf024f8.cer: not found in local repository
May 18 23:03:33 run-validation: rsync://repository.lacnic.net/rpki/lacnic/48f083bb-f603-4893-9990-0284c04ceb85/31ef1c6ecf5c293c01f2c63f43f6ba98ebf024f8.cer: failed to load.