[dns-stats-users] Start of month XML import error
John Dickinson
jad at sinodun.com
Mon Jun 8 10:56:57 UTC 2015
Hi,
Thanks for the report.
I think this may be due to your machine being set to BST not UTC.
This has come up a few times and I will look into making the
importer/database more intelligent.
regards
John
On 08/06/2015 11:45, matt singh wrote:
> Hello,
> I have noticed that import of xml data for my node has not been
> working since the beginning of this month.
>
> If I try and re-import old data using:
> sudo -u hedgehog /usr/local/bin/refile_and_grok.sh -s 2015-06-01
>
> dsc-extractor-xml-db.out
>
> ----------------------- 2015-06-08 11:08:25 BST ----------------------
> *** Database connected
> *** Processing server: ns_5
> *** Processing node: Node-1
> *** Found: 10684 .xml files to process.
>
> 2015-06-08 11:08:25 BST: Starting: ./incoming/1433116920.dscdata.xml
> Format 3b write_db runtime error: ERROR: new row for relation
> "data_ns_5_idn_vs_tld_2015_06" violates check constraint
> "data_ns_5_idn_vs_tld_2015_06_starttime_check"
> DETAIL: Failing row contains (2015-06-01 00:00:00+01, 2, 2, 16, ALL,
> abc, 5).
>
> The psql table is indeed empty.
> select * from data_ns_5_idn_vs_tld_2015_06;
> starttime | server_id | node_id | plot_id | key1 | key2 | value
> -----------+-----------+---------+---------+------+------+-------
> (0 rows)
>
>
> Can I only import historical data if the file format is DAT?
>
> I've converted some of the files to dat format using the dsc-extractor
> utility. But got a similar error when running
> sudo -u hedgehog /usr/local/bin/refile_and_grok.sh -i DAT
>
> ----------------------- 2015-06-08 11:41:46 BST ----------------------
> *** Database connected
> *** Processing server: ns_5
> *** Processing node: Node-1
> *** Found: 22 .dat files to process.
>
> 2015-06-08 11:41:46 BST: Starting: ./20150601/certain_qnames_vs_qtype.dat
> Format 2 unique_violation error: ERROR: duplicate key value violates
> unique constraint "pk_data_ns_5_certain_qnames_vs_qtype_2015_06"
> DETAIL: Key (server_id, node_id, plot_id, starttime, key1, key2)=(2,
> 2, 1, 2015-06-01 02:55:00+01, else, 15) already exists.
>
> 2015-06-08 11:41:47 BST: Error: failed to process:
> ./20150601/certain_qnames_vs_qtype.datReason: Failed to process data
> 2015-06-08 11:41:47 BST: File renamed to:
> ./20150601/certain_qnames_vs_qtype.dat.failed.7083-7868-9f17-1423
> 2015-06-08 11:41:47 BST: Completed:
> ./20150601/certain_qnames_vs_qtype.dat 21 remaining (1
> failures).
>
> 2015-06-08 11:41:47 BST: Starting:
> ./20150601/client_addr_vs_rcode_accum.dat
> Format 4 write_db Runtime error: ERROR: new row for relation
> "data_ns_5_client_addr_vs_rcode_accum_2015_06" violates check
> constraint "data_ns_5_client_addr_vs_rcode_accum_2015_06_starttime_check"
> DETAIL: Failing row contains (2015-06-01 00:00:00+01, 2, 2, 3,
> 201.150.62.201, 0, 28).
>
> Any ideas what I'm overlooking here?
>
> thanks,
> matt
>
> _______________________________________________
> dns-stats-users mailing list
> dns-stats-users at dns-stats.org
> https://mm.dns-stats.org/mailman/listinfo/dns-stats-users
More information about the dns-stats-users
mailing list