[dns-stats-users] Start of month XML import error

[matt singh]

Hello,
I have noticed that import of xml data for my node has not been working 
since the beginning of this month.

If I try and re-import old data using:
sudo -u hedgehog /usr/local/bin/refile_and_grok.sh -s 2015-06-01

dsc-extractor-xml-db.out

----------------------- 2015-06-08 11:08:25 BST ----------------------
*** Database connected
*** Processing server: ns_5
*** Processing node:   Node-1
*** Found:             10684 .xml files to process.

2015-06-08 11:08:25 BST: Starting:  ./incoming/1433116920.dscdata.xml
Format 3b write_db runtime error: ERROR:  new row for relation 
"data_ns_5_idn_vs_tld_2015_06" violates check constraint 
"data_ns_5_idn_vs_tld_2015_06_starttime_check"
DETAIL:  Failing row contains (2015-06-01 00:00:00+01, 2, 2, 16, ALL, 
abc, 5).

The psql table is indeed empty.
select * from data_ns_5_idn_vs_tld_2015_06;
  starttime | server_id | node_id | plot_id | key1 | key2 | value
-----------+-----------+---------+---------+------+------+-------
(0 rows)


Can I only import historical data if the file format is DAT?

I've converted some of the files to dat format using the dsc-extractor 
utility. But got a similar error when running
sudo -u hedgehog /usr/local/bin/refile_and_grok.sh -i DAT

----------------------- 2015-06-08 11:41:46 BST ----------------------
*** Database connected
*** Processing server: ns_5
*** Processing node:   Node-1
*** Found:             22 .dat files to process.

2015-06-08 11:41:46 BST: Starting:  ./20150601/certain_qnames_vs_qtype.dat
Format 2 unique_violation error: ERROR:  duplicate key value violates 
unique constraint "pk_data_ns_5_certain_qnames_vs_qtype_2015_06"
DETAIL:  Key (server_id, node_id, plot_id, starttime, key1, key2)=(2, 2, 
1, 2015-06-01 02:55:00+01, else, 15) already exists.

2015-06-08 11:41:47 BST: Error: failed to process: 
./20150601/certain_qnames_vs_qtype.datReason:  Failed to process data
2015-06-08 11:41:47 BST: File renamed to: 
./20150601/certain_qnames_vs_qtype.dat.failed.7083-7868-9f17-1423
2015-06-08 11:41:47 BST: Completed: 
./20150601/certain_qnames_vs_qtype.dat              21 remaining (1 
failures).

2015-06-08 11:41:47 BST: Starting: 
./20150601/client_addr_vs_rcode_accum.dat
Format 4 write_db Runtime error: ERROR:  new row for relation 
"data_ns_5_client_addr_vs_rcode_accum_2015_06" violates check constraint 
"data_ns_5_client_addr_vs_rcode_accum_2015_06_starttime_check"
DETAIL:  Failing row contains (2015-06-01 00:00:00+01, 2, 2, 3, 
201.150.62.201, 0, 28).

Any ideas what I'm overlooking here?

thanks,
matt